Description: Flexera One - IT Asset Management - NA - Inventory Processing Disruption

Timeframe: July 19th, 5:30 PM to July 21st, 2:00 AM PDT

‌

Incident Summary

On July 19th, 2024, at 5:30 PM PDT, we experienced a disruption in our IT Asset Management Platform, specifically affecting inventory processing in the NA region. Customers may have experienced delays in inventory processing and potential backlogs during this disruption. Additionally, affected client beacons may have encountered delays in data updates. Inventory processing resumed around 2:00 AM PDT on July 21st, following a failover event to one of the clusters.

Preliminary investigations indicated an issue with one of our data processing interfaces. Monitoring revealed a substantial increase in error responses and target response times coinciding with the disruption. Usage data, which bypasses this interface, continued to flow normally.

Our current monitors did not alert us to the issue as they were just above the threshold due to the normal flow of usage data. This oversight led to a backlog and delay in processing inventory files.

We continued to monitor the environment closely. On July 21st at 6:00 PM PDT, additional health checks confirmed that inventory processing and data handling were consistently at normal levels. After validation from all involved teams, the incident was resolved and transitioned to a problem management investigation.

Subsequent investigation revealed that the primary cause of the disruption was a critical server responsible for performing
authentication requests experiencing a failure due to a storage issue. Additionally, the load balancer handling data redirection
encountered significant errors, including numerous 502 errors and increased response times, further contributing to the disruption.

Root Cause

Primary Root Cause

The primary cause of the disruption was a critical server responsible for performing authentication requests experiencing a failure due to a storage issue.

Contributing Factors

• Load Balancer Errors: The load balancer handling data redirection encountered significant errors, including numerous 502 errors and increased response times, which further contributed to the disruption.
• Monitoring Thresholds: Current monitoring thresholds did not trigger alerts as they were just above the threshold due to the normal flow of usage data, leading to a delay in identifying the issue.

Remediation Actions

1. Cluster Failover: The system initiated a failover event to an alternate cluster, which restored inventory processing functionality.
2. Continuous Monitoring: Monitored the environment closely to ensure that inventory processing and data handling were stable and consistently at normal levels.
3. Health Checks: Conducted additional health checks to validate the stability and performance of the system after the failover.
4. Disk Space Management: Identified the storage issue on the critical server and created a task to increase the disk space to prevent similar failures in the future.

Future Preventative Measures

1. Disk Space Management Enhancements: We have increased the disk size to address storage issues. This proactive measure ensures that sufficient storage capacity is available to support ongoing operations and reduces the likelihood of similar disruptions in the future.
2. Enhanced Monitoring and Alerting: Our technical teams have been tasked with implementing a comprehensive monitoring and alerting system to detect high disk usage and other critical metrics across services. This system will ensure timely alerts are sent to the relevant teams, enabling them to address potential issues before they impact operations.