Description: Flexera One - IT Asset Management - NA - Inventory Processing Disruption
Timeframe: July 19th, 5:30 PM to July 21st, 2:00 AM PDT
Incident Summary
On July 19th, 2024, at 5:30 PM PDT, we experienced a disruption in our IT Asset Management Platform, specifically affecting inventory processing in the NA region. Customers may have experienced delays in inventory processing and potential backlogs during this disruption. Additionally, affected client beacons may have encountered delays in data updates. Inventory processing resumed around 2:00 AM PDT on July 21st, following a failover event to one of the clusters.
Preliminary investigations indicated an issue with one of our data processing interfaces. Monitoring revealed a substantial increase in error responses and target response times coinciding with the disruption. Usage data, which bypasses this interface, continued to flow normally.
Our current monitors did not alert us to the issue as they were just above the threshold due to the normal flow of usage data. This oversight led to a backlog and delay in processing inventory files.
We continued to monitor the environment closely. On July 21st at 6:00 PM PDT, additional health checks confirmed that inventory processing and data handling were consistently at normal levels. After validation from all involved teams, the incident was resolved and transitioned to a problem management investigation.
Subsequent investigation revealed that the primary cause of the disruption was a critical server responsible for performing
authentication requests experiencing a failure due to a storage issue. Additionally, the load balancer handling data redirection
encountered significant errors, including numerous 502 errors and increased response times, further contributing to the disruption.
Root Cause
Primary Root Cause
The primary cause of the disruption was a critical server responsible for performing authentication requests experiencing a failure due to a storage issue.
Contributing Factors
• Load Balancer Errors: The load balancer handling data redirection encountered significant errors, including numerous 502 errors and increased response times, which further contributed to the disruption.
• Monitoring Thresholds: Current monitoring thresholds did not trigger alerts as they were just above the threshold due to the normal flow of usage data, leading to a delay in identifying the issue.
Remediation Actions
Future Preventative Measures