Flexera System Status Dashboard Status - Flexera One - IT Visibility - Technology Intelligence Platform - NAM,APAC & EU- Issue with loading PowerBI reports

Flexera One - IT Visibility - Technology Intelligence Platform - NAM,APAC & EU- Issue with loading PowerBI reports

Incident Report for Flexera System Status Dashboard

Postmortem

Description: Flexera One - Technology Intelligence Platform - NAM, APAC & EU- Issue with loading PowerBI reports

Timeframe: October 8, 2024, 5:29 AM PDT to October 8 2024, 8:31 AM PDT

Incident Summary

On Tuesday, October 8th, 2024, at 5:29 AM PDT, our teams detected a service disruption affecting the Technology Intelligence Platform. The disruption impacted customers across the NAM, APAC, and EU regions, preventing them from viewing data in PowerBI reports. Initial investigations found no issues with the PowerBI backend services. However, by 5:59 AM PDT, our teams identified the root cause as a recent UI deployment that was affecting the report rendering. The team immediately initiated a rollback, which was successfully completed by 8:30 AM PDT. After confirming that all services were fully restored, we declared the issue resolved at 8:31 AM PDT.

‌

Root Cause

Our team identified that a recent change had been made to a UI component to improve accessibility within the platform. The change inadvertently caused PowerBI reports to fail when loading, as the UI adjustment interfered with data rendering.

‌

Remediation Actions

Once the root cause was identified, the technical team promptly rolled back the UI change across all regions. This rollback restored normal functionality, allowing customers to once again access and view PowerBI reports. Post-rollback health checks confirmed that all affected services were fully operational, and the incident was resolved.

‌

Future Preventative Measures

· CI/CD Pipeline Updates: The UI CI/CD pipeline has been updated to enable swift recovery. Future reverts or rollbacks can now be executed within minutes, particularly when reverting to a version that has already been tested and successfully deployed in live environments.

· Enhanced Testing for UI Changes: Additional layers of regression and impact testing will be applied specifically to UI changes, particularly those that affect data visualization components like PowerBI.

· Monitoring Improvements: Improved monitoring of frontend and backend interactions will be implemented to ensure earlier detection of potential issues resulting from UI changes.

Posted Oct 23, 2024 - 06:11 PDT

Resolved

We have reverted the change across all regions to restore functionality. Following the rollback, our health checks have confirmed that all affected services are now fully operational. This incident has been resolved.

Posted Oct 08, 2024 - 08:36 PDT

Identified

Our investigations have identified a recent update as the cause of the disruption. Our technical team is currently working on reverting the change. We will keep you informed with updates as progress is made.

Posted Oct 08, 2024 - 07:20 PDT

Investigating

Incident Description:
We have detected a service disruption affecting our Technology Intelligence Platform, which may be impacting customers in the NAM, APAC, and EU regions. As a result, affected users are currently unable to load PowerBI reports.

Priority: P2

Restoration activity:
Our technical teams are actively investigating this issue. They are working to restore the services and identify the underlying root cause .

Posted Oct 08, 2024 - 05:48 PDT

This incident affected: Flexera One - IT Visibility - North America (IT Visibility US), Flexera One - IT Visibility - Europe (IT Visibility EU), and Flexera One - IT Visibility - APAC (IT Visibility - APAC).