Flexera One - IT Visibility - NAM - Inventory Data Out of date

Incident Report for Flexera System Status Dashboard

Postmortem

Description: Flexera One - IT Visibility - NAM - Inventory Data Out of date

Timeframe: July 22, 2025, 1:05 AM PDT  to July 26, 2025, 5:27 PM PDT

Incident Summary

 

On Tuesday, July 22nd, 2025, at 1:09 AM PDT, an issue was detected that resulted in delays in the processing of inventory data for IT Visibility services in the North America region. Although the Flexera One platform and IT Visibility remained accessible, the uploads of inventory data did not process as anticipated, leading to a lack of updated inventory information within the system. This situation resulted in a temporary lack of visibility into the most recent inventory data for customers.

Despite the upload jobs appearing to be successful, they either timed out or were recorded as empty. As a result, the data displayed in the user interface and reports was outdated, reflecting a delay ranging from approximately 30 minutes to several hours, depending on the time of access. Furthermore, Technopedia services may have also experienced impacts during this disruption. It is noteworthy that no customer reports were received during the incident window.

In response, our teams executed a full data reload and restarted the inventory processing service to initiate recovery efforts. By July 22, 2025, at 12:36 PM PDT, the affected services were restored, and our teams began addressing the backlog that had accumulated during the outage. Our teams continued to monitor the backlog closely and officially declared the issue resolved after the backlog was fully processed on July 26, 2025, at 5:27 PM PDT.

Root Cause

 

Investigations by our technical teams revealed that the disruption was caused by automated patch updates from our cloud service provider, which unexpectedly impacted core components responsible for inventory data ingestion and other critical services. These services are essential for processing uploaded inventory data into the platform.

The automated patches led to unexpected service behaviour, where data ingestion services did not fail visibly but caused silent interruptions in downstream processing.

Remediation Actions 

 

·        System Recovery Initiated: Our teams performed a full data reload and restarted the inventory processing service to begin recovery efforts.

·        Core Services Restored: The key services responsible for handling inventory data were successfully brought back online, allowing delayed data to resume processing.

·        Data Accuracy Ensured: To ensure no loss of critical information, our teams initiated a replay of agent data, covering the duration of the disruption.

·        Progressive Backlog Reduction: Over the following days, our teams closely monitored processing progress and worked to reduce the backlog from two days to less than ten hours.

·        Full Restoration Achieved: As of July 27, 2025, 12:59 PM PDT, processing had fully caught up, and normal operations resumed. The incident was closed after confirming all systems were stable and up to date.

Future Preventative Measures 

 

·        Enhanced Monitoring: Deploy additional monitoring metrics and alerts specifically for normalization steps.

·        Automated Health Checks: Introduce periodic end-to-end health checks of the service to proactively detect processing stalls.

·        Resilience Improvements: Explore service-level redundancy and retry mechanisms to allow ingestion and other critical services to recover autonomously from transient patch-induced disruptions.

Posted Aug 07, 2025 - 04:12 PDT

Resolved

Backlog processing was successfully completed on 26th July at 4:27 PM PST. All data is now fully up to date.
Posted Jul 27, 2025 - 00:33 PDT

Update

Inventory data processing continues to progress, and the backlog has been significantly reduced. The system is approaching full catch-up, and we are monitoring closely. We will provide confirmation once data is fully current.
Posted Jul 26, 2025 - 09:54 PDT

Update

Inventory data processing continues to progress steadily. Some data may still appear delayed by less than a day as the system works through the remaining backlog. We are monitoring closely and will provide confirmation once data is fully current.
Posted Jul 25, 2025 - 16:49 PDT

Update

Inventory data continues to be processed, and updates are appearing across customer environments. Earlier in the recovery process, an optimization was applied to quickly move past unaffected historical data and reach the point of impact. The system is now processing that relevant data in full, which is more resource-intensive and requires additional time to complete.

Some data may still be delayed by up to two days during this final stage of catch-up. We are continuing to monitor closely and will provide confirmation once data processing is fully current.
Posted Jul 24, 2025 - 16:50 PDT

Update

Backlog processing remains in progress and continues to trend in the right direction. We are monitoring closely and expect to see continued improvement throughout the day. Further updates will be shared as progress continues.
Posted Jul 24, 2025 - 08:55 PDT

Update

Inventory data processing is advancing, and earlier delays are being reduced as backlog processing continues. We are keeping a close watch on system performance and will provide updates as progress is made.
Posted Jul 23, 2025 - 16:13 PDT

Update

Inventory data processing continues to make progress. Most data sources have caught up, while a portion of agent-based inventory data is still being processed. This may take additional time to fully complete. Our teams continue to monitor closely and will provide further updates as needed.
Posted Jul 23, 2025 - 11:07 PDT

Update

Inventory data processing has largely returned to expected levels. A small portion of data is still catching up and is expected to complete shortly as processing continues.

We are also working through a backlog of agent-based inventory data, which may take additional time to process fully. Our teams are actively optimizing the process to help accelerate completion.
Posted Jul 22, 2025 - 18:21 PDT

Monitoring

Inventory data processing has resumed. The platform remains fully accessible, though it may take a few hours for all data to reflect the most recent updates. Our teams are monitoring progress closely to ensure timely completion.
Posted Jul 22, 2025 - 12:41 PDT

Update

The full data reload has been completed. Technical teams are now working on restarting inventory data processing and will continue to monitor progress closely.
Posted Jul 22, 2025 - 12:31 PDT

Identified

Our teams have confirmed that the issue was caused by automated patch updates from our cloud service provider. A fix has been implemented, and the teams are currently working on reloading the files. Due to the large volume of data, this process may take some time. We will share further updates as they become available.
Posted Jul 22, 2025 - 05:37 PDT

Investigating

Incident Description: We are currently addressing an issue impacting inventory data processing for IT Visibility services in the NA region. While the system remains accessible, inventory data uploads have been delayed, which may affect the availability of updated inventory information. The issue might also impact Technopedia services.

Priority: P2

Restoration Activity: Our technical team is diligently working to identify the root cause and restore services. Initial investigations indicate that the issue may be related to automated patch updates from our cloud service provider. Our teams are continuing to investigate this matter and monitor the platform closely. We will provide further updates as we make progress .
Posted Jul 22, 2025 - 00:50 PDT
This incident affected: Flexera One - IT Visibility - North America (IT Visibility US).