Cloud Spend Data in Optima and Flexera One Not Available.
Incident Report for Flexera System Status Dashboard
Postmortem

Description: Cloud Spend Data in Optima and Flexera One Not Available

Timeframe: June 5th 19:30 PDT to June 5th 23:39 PDT

Incident Summary

On Saturday 5th June 19:30 PDT customers using Optima and Flexera One were unable to view their Cloud Spend data.

Technical teams were alerted to high error rates by monitoring systems automatically and responded promptly - Investigations found some services were intermittently failing.

At 20:21 PDT additional subject matter experts were engaged to assist with the investigation.

At 20:57 PDT during the investigations, technical staff found that a data purge process was paused for a key UI service. This resulted in additional load on multiple services which rendered them intermittently unresponsive. Data purge processes were resumed restoring services.

At 21:53 PDT it was confirmed Cloud Spend data was visible to all customers in Optima and Flexera One. After additional monitoring, technical staff confirmed all services to be running normally and the incident was declared resolved at 23:39 PDT.

Root Cause

After a thorough investigation, it was found that the task to purge obsolete data was unintentionally paused for one of the services responsible for rendering data in UI. This caused excess data accumulation and the resulting load on the UI service rendered it unresponsive. This resulted in Cloud Spend data not being visible to customers.

Corrective Action

• Appropriate checks have been added to the monitoring systems to alert when data purge activities are halted.
• Additional resources have been assigned to the services to handle temporary loads as required.

Posted Jul 08, 2021 - 03:44 PDT

Resolved
Monitoring has confirmed services are stable. This Incident is now resolved.
Posted Jun 05, 2021 - 23:39 PDT
Monitoring
Technical staff have restored visibility of all Customer Cloud spend/cost data. We are monitoring services to ensure they remain stable.
Posted Jun 05, 2021 - 21:53 PDT
Update
Technical staff are progressively restoring services, customer Cloud spend/cost data is continuing to return to visibity in the UI.
Posted Jun 05, 2021 - 20:57 PDT
Update
We are continuing to work on a fix for this issue.
Posted Jun 05, 2021 - 20:21 PDT
Identified
Incident Description:

Cloud spend/cost data is currently unavailable in Optima and Flexera One at the moment. We are working to restore this functionality.
Posted Jun 05, 2021 - 20:19 PDT
This incident affected: Flexera One - Cloud Management - North America (Cloud Cost Optimization - US).