Flexera One - Cloud Cost Optimization (CCO) - APAC, NAM & EU- Billing data missing

Incident Report for Flexera System Status Dashboard

Postmortem

Description: Flexera One - Cloud Cost Optimization (CCO) - APAC, NAM & EU- Billing data missing

Timeframe: May 20, 2025, 6:43 PM PST to May 22, 2025, 8:00 PM PST

‌

Incident Summary

On Tuesday, May 20 at 6:43 PM PST, the Cloud Cost Optimization (CCO) platform began experiencing issues that impacted some customers, primarily in the APAC region. While the platform itself remained accessible, affected users noticed missing billing and cost data, along with disruptions to related optimizations.

The issue was detected early by internal monitoring and triaged promptly. Initial investigations pointed to a recent update deployed to the analytics database import process earlier that day. Although the platform's core functionality was unaffected, the absence of accurate data rendered some features ineffective for customers.

‌

Root Cause

The root cause of the incident was a flaw introduced in a recent update to the analytics database import process. The update led to incomplete or missing data for some customer organizations after the scheduled import jobs. Because imports occur on a delayed cycle, the issue was not immediately visible during post-deployment testing.

This uncovered a gap in testing coverage related to the lack of validations that occur after an analytics database import has been run.

‌

Remediation Actions

· Fix Deployment: A corrective update was developed and deployed by May 20 at 9:09 PM PST to address the faulty analytics database import process.

· Manual Data Imports: Teams initiated manual analytics database imports to expedite data correction for all affected organizations.

· Regional Validation Completion: By May 21 at 1:44 AM PST, APAC and EU region data validations were completed, confirming service restoration in those regions.

· NAM Region Follow-up: The team shifted focus to the NAM region, resolving most cases by May 21 at 10:48 AM PST, with only a few remaining.

· Ongoing Monitoring: On May 21 at 3:13 PM PST, all NAM organizations were successfully processed and the incident was moved to the Monitoring phase.

· Final Resolution Confirmation: On May 22 at 8:00 AM PST, full service restore was confirmed post extended monitoring .

‌

Remediation Actions

· Fix Deployment: A corrective update was developed and deployed by May 20 at 9:09 PM PST to address the faulty analytics database import process.

· Manual Data Imports: Teams initiated manual data imports to expedite data correction for all affected organizations.

· Regional Validation Completion: By May 21 at 1:44 AM PST, APAC and EU region data validations were completed, confirming service restoration in those areas.

· NAM Region Follow-up: The team shifted focus to the NAM region, resolving most cases by May 21 at 10:48 AM PST, with only a few remaining.

· Ongoing Monitoring: On May 21 at 3:13 PM PST, all NAM organizations were successfully processed the incident was moved to the Monitoring phase.

· Final Resolution Confirmation: On May 22 at 8:00 AM PST, full service restore was confirmed post extended monitoring .

Posted May 30, 2025 - 03:12 PDT

Resolved

Our teams have closely monitored the platform over an extended period and have observed no recurrence of the issue. We have declared the incident as resolved.

Posted May 21, 2025 - 21:05 PDT

Update

The environment remains stable, and we have not observed any new issues. Our teams will continue monitoring over the next few hours to ensure continued stability.

Posted May 21, 2025 - 16:16 PDT

Update

Most organizations have been brought up to date. A small number of remaining organizations are still being addressed, and our teams are working directly to resolve those.

We will continue to monitor progress closely and share further updates as needed.

Posted May 21, 2025 - 10:43 PDT

Update

Following resolution in the APAC and EU regions, we continue to make progress restoring billing data for impacted organizations in the NAM region. The work is being carried out in phases due to the scale of the environment.

We will continue to provide updates as restoration efforts progress.

Posted May 21, 2025 - 07:55 PDT

Update

All validations for the APAC and EU regions have been successfully completed, and services have been fully restored. Our teams are now focusing on the NAM region, which will require an extended restoration window due to the scale of infrastructure involved. We will share further updates as they become available.

Posted May 21, 2025 - 00:36 PDT

Update

Following the deployment of the fix, our team performed additional validations and manually restarted several services. Services have now been restored for the majority of affected customers. We continue to closely monitor the platform and are working toward full restoration for all users.

Posted May 21, 2025 - 00:12 PDT

Monitoring

A fix has been implemented and we are monitoring the results.

Posted May 20, 2025 - 20:48 PDT

Identified

Our team has identified the root cause of the issue as a recent update pushed earlier today. A fix has been developed and is currently being prepared for deployment.

Posted May 20, 2025 - 19:58 PDT

Update

We are continuing to investigate this issue.

Posted May 20, 2025 - 19:24 PDT

Investigating

Incident Description: We are currently experiencing an issue within the Cloud Cost Optimization (CCO) platform affecting some customers in the APAC, NAM & EU regions. While the platform remains accessible; affected customers may notice missing billing and cost data, as well as disruptions to related optimizations.

Priority: P2

Restoration Activity: Our technical team is actively investigating the root cause and working to restore services as quickly as possible. We will continue to share updates as we make progress toward a resolution.

Posted May 20, 2025 - 18:41 PDT

This incident affected: Flexera One - Cloud Management - Europe (Cloud Cost Optimization - EU), Flexera One - Cloud Management - APAC (Cloud Cost Optimization - APAC), and Flexera One - Cloud Management - North America (Cloud Cost Optimization - US).