Flexera One - IT Asset Management - EU - Inventory Data Upload Delayed

Description: Flexera One - IT Asset Management - EU - Inventory Data Upload Delayed

Timeframe: November 24th, 4:41 AM to November 24th, 6:26 AM PST

Incident Summary

On Friday, November 24th, at 4:41 AM PST, we received reports of customers experiencing difficulties accessing inventory data on the IT Asset Management platform. This may have caused delays for some customers, particularly in the EU region, while other regions remained unaffected.

Our technical teams responded promptly to resolve the issue. After a thorough investigation, it was identified that the problem may have arisen from one of the tenant IDs initiating an operation that failed to finish, resulting in a backlog of all other inventory data.

Upon scrutinizing the operation executed by the tenant ID and assessing the potential effects associated with canceling it, our technical team successfully terminated the operation at 4:57 AM PST. Subsequently, health checks began to pass successfully, and blocked sessions were cleared.

By 5:11 AM PST, the database had returned to a healthy state. However, ongoing monitoring was maintained to ensure the successful processing of the backlog. At 6:26 AM PST, following successful health checks, we reached out to our affected customers to verify the resolution of the incident. Upon confirmation, the incident was considered resolved.

Root Cause

The issue originated from an operation initiated by one of the tenant IDs that did not complete as expected, resulting in a buildup of backlog of subsequent inventory data in the database.

Remediation Actions

1. Identification and Termination: A comprehensive investigation identified an incomplete operation initiated by a tenant ID, leading to a successful termination at 4:57 AM PST.
2. Ongoing Monitoring: Continuous monitoring was maintained to ensure the smooth processing of the backlog and prevent recurrence.
3. Verification with Customers: At 6:26 AM PST, following successful health checks, we reached out to our affected customers to verify the resolution of the incident.

Future Preventative Measures

1. Proactive Enhancement of Inventory Import Architecture: We acknowledge the identified issue that prompted our focus on enhancement to our inventory import architecture. Consequently, we have recently implemented a change to the cloud, transitioning to a new queue and ensuring that one tenant's operations do not impede others. This change is aimed at significantly alleviating the identified issue.
2. Enhanced Alerting Strategy for Monitoring: While we currently have monitoring in place, we recognize the need for an effective alerting mechanism. We are committed to revising and adapting our monitoring and alerting strategies to proactively address this concern.