Description: Flexera One - IT Asset Management - EU - Inventory Data Processing Delayed

Timeframe: November 26, 2024, 12:59 AM PDT – December 1, 2024, 3:36 PM PDT

Incident Summary

On November 26, 2024, at 12:59 AM PDT, the IT Asset Management (ITAM) platform in the EU region experienced delays in inventory data processing. While the platform remained accessible, data uploads related to inventory files were delayed, impacting the availability of updated records for some customers.

The issue was traced to a halted service responsible for processing incoming inventory files, causing disruptions in data processing. As a result, some users may have experienced slower load times or temporary delays in accessing updated data during this period.

Upon identifying the root cause, our technical teams restarted the affected service on November 29, 2024, at 2:59 AM PDT, enabling inventory file processing to resume. Real-time monitoring was immediately implemented to track progress and ensure system stability.

The backlog reduction continued steadily over the following days. Full recovery was achieved on December 1, 2024, at 3:36 PM PDT, with all inventory files successfully processed and system performance restored to normal.

Root Cause

Primary Root Cause

• Service Failure: A key processing service encountered an unexpected error while handling incoming data files. The exact cause of this failure is still being investigated, though limited system memory is suspected as a contributing factor.

Contributing Factors

• Undetected Service Stagnation: The service remained active but stopped processing files, delaying detection since no system-level alerts were triggered.
• Delayed Detection: Existing monitoring and alerting mechanisms did not detect the service failure promptly, resulting in a delayed response and reliance on manual notifications.
• System Load: The continuous inflow of new files added to the backlog, increasing system load and processing time.

Remediation Actions

1. Service Restart: The affected service was restarted, restoring normal file processing.
2. Enhanced Monitoring: Custom monitoring dashboards were set up to track data processing and ensure continuous file uploads.
3. Data Prioritization: Data from impacted customers was prioritized to reduce service delays as quickly as possible.
4. Team Collaboration: Cross-functional teams collaborated closely to identify the issue, implement a resolution, and monitor recovery progress.

Future Preventative Measures

Implemented Measures

1. Improved Monitoring and Alerting: New monitoring rules have been added at the application level to detect service disruptions even when the service remains active but stops processing files.
2. Incident Management Integration: Critical alerts are now integrated into the incident management system, ensuring timely notifications through multiple channels.
3. Microservice for Backlog Insights: A dedicated microservice was developed to provide deeper visibility into file backlogs, allowing teams to identify and resolve issues more effectively.

Planned Measures (Q1 2024)

1. Root Cause Investigation: A detailed technical review is underway to identify the exact cause of the service failure and potential resource limitations. System improvements will be applied based on the findings.
2. Monitoring and Alerting Overhaul: The monitoring and alerting infrastructure is undergoing a comprehensive review to address detection gaps. This effort focuses on implementing advanced monitoring tools, enhancing metrics, and conducting regular audits to maintain long-term system reliability.
3. Incident Response Readiness: Runbooks for managing production incidents are being developed based on past incidents, ensuring a well-documented response process for future events.
4. Metrics Optimization: Existing system metrics will be continuously reviewed and refined to improve alert accuracy and operational insights.