Description: Cloud Spend Data in Optima and Flexera One Not Available
Timeframe: June 5th 19:30 PDT to June 5th 23:39 PDT
On Saturday 5th June 19:30 PDT customers using Optima and Flexera One were unable to view their Cloud Spend data.
Technical teams were alerted to high error rates by monitoring systems automatically and responded promptly - Investigations found some services were intermittently failing.
At 20:21 PDT additional subject matter experts were engaged to assist with the investigation.
At 20:57 PDT during the investigations, technical staff found that a data purge process was paused for a key UI service. This resulted in additional load on multiple services which rendered them intermittently unresponsive. Data purge processes were resumed restoring services.
At 21:53 PDT it was confirmed Cloud Spend data was visible to all customers in Optima and Flexera One. After additional monitoring, technical staff confirmed all services to be running normally and the incident was declared resolved at 23:39 PDT.
After a thorough investigation, it was found that the task to purge obsolete data was unintentionally paused for one of the services responsible for rendering data in UI. This caused excess data accumulation and the resulting load on the UI service rendered it unresponsive. This resulted in Cloud Spend data not being visible to customers.
• Appropriate checks have been added to the monitoring systems to alert when data purge activities are halted.
• Additional resources have been assigned to the services to handle temporary loads as required.