Flexera One - IT Asset Management - Inventory Data delayed
Incident Report for Flexera System Status Dashboard
Postmortem

The full post-mortem report is available here: https://status.flexera.com/incidents/jdqfv3j4x71c

Posted May 13, 2024 - 17:16 PDT

Resolved
This incident has been resolved.
Posted Mar 27, 2024 - 08:11 PDT
Update
The NDI backlog is currently at satisfactory levels. However, we are closely monitoring the situation to ensure seamless processing.
Posted Mar 26, 2024 - 09:36 PDT
Update
We are continuing to observe a reduction in the backlog, reflecting ongoing improvements in the system. We are continuing to monitor the progress closely.
Posted Mar 25, 2024 - 19:33 PDT
Update
The NDI backlog has seen a notable reduction over the weekend, attributed to the recent server isolation initiatives. The technical teams are closely monitoring for further progress.
Posted Mar 25, 2024 - 07:39 PDT
Update
We are actively implementing measures to expedite processing, including segregating UsageData uploads onto dedicated servers to enhance efficiency.

As a result of these adjustments, our team has observed significant progress, with a further reduction in the backlog.
Posted Mar 22, 2024 - 14:17 PDT
Update
We are making progress in resolving inventory backlogs, with recent fixes leading to notable improvements in our backlog management. We will continue to provide updates on our progress.
Posted Mar 21, 2024 - 11:18 PDT
Monitoring
We have identified instances of backlog rising. To address this, we have implemented a hotfix that includes scaling up server resources. We are closely monitoring the situation for further developments.
Posted Mar 20, 2024 - 20:18 PDT
Update
We have been continuing to monitor the environment and as a result of the actions implemented, the current processing of inventory indicates improved timeliness, with a reduced gap between the file date and processing time compared to previous instances.
Posted Mar 20, 2024 - 06:49 PDT
Update
We have observed a recurrence of a notable backlog in UsageData processing. We have taken additional measures, and our team is actively working to address this issue and bring the environment back to its stable state.
Posted Mar 19, 2024 - 11:19 PDT
Update
Our team continues to monitor and manually intervene to process a significant backlog, compounded by additional incoming data. As part of our efforts to expedite progress, we have implemented measures to streamline the process. Further updates will follow as developments unfold.
Posted Mar 19, 2024 - 02:18 PDT
Update
Our team continues to monitor and manually intervene to process a significant backlog, compounded by additional incoming data. As part of our efforts to expedite progress, we have implemented measures to streamline the process. Further updates will follow as developments unfold.
Posted Mar 18, 2024 - 18:16 PDT
Update
We are actively monitoring the backlog's progress, which has been proceeding smoothly thus far. We continue to closely monitor the progress and will take additional actions to expedite it as required.
Posted Mar 18, 2024 - 13:48 PDT
Update
We have made progress with the introduction of a new inventory shard, effectively managing both new inventories and the backlog. However, we are still exploring additional steps to expedite processing. Discussions are ongoing to ensure the most effective approach. We will continue to keep you updated.
Posted Mar 18, 2024 - 06:40 PDT
Update
While we are observing a reduction in the inventory backlog, further analysis is ongoing to explore measures to enhance and streamline the processing. Additionally, we suspect that heavy imports over the weekend may have contributed to the backlog. As discussions continue, we are closely monitoring progress and will provide updates accordingly.
Posted Mar 17, 2024 - 16:01 PDT
Update
We are closely monitoring the inventory backlog to identify any obstacles to efficient processing. Simultaneously, we're exploring long-term solutions and alternative approaches to address these challenges effectively. We'll provide further updates as developments unfold.
Posted Mar 17, 2024 - 10:26 PDT
Update
Our technical teams have made further progress. With the implementation of additional measures, our services are now operating optimally, prompting the resumption of halted data processing. Consequently, we have lifted the restriction, allowing the processing of new data over the weekend.
Posted Mar 15, 2024 - 14:37 PDT
Update
Our investigation continues, and we have made substantial progress in addressing challenges with usage data uploads. External factors affecting infrastructure components have been identified, leading to a buildup of usage data. Efforts have been made to resolve this, including clearing backlog files that may have contributed to the issue. However, despite these efforts, usage data levels in our monitoring systems haven't shown any significant decreases.

Our technical team has opted to wait a bit longer before considering removing usage data traffic restrictions implemented earlier. We remain committed to closely collaborating with our partners to monitor and address this issue effectively, providing updates as necessary.
Posted Mar 15, 2024 - 10:56 PDT
Update
Technical teams continue to investigate the root cause and potential solutions - backlog processing is now making progress.
Posted Mar 14, 2024 - 20:21 PDT
Update
Our technical team is still observing a backlog that hasn't decreased as anticipated despite the implemented changes. Investigations are ongoing, and multiple plans have been proposed. We are currently evaluating the most suitable course of action to promptly address the situation.
Posted Mar 14, 2024 - 17:39 PDT
Identified
We have continued to monitor the environment overnight. Our ongoing investigations indicate that the issue stems from a significant volume of usage data files originating from a single source. To address this, our team has effectively isolated the usage data traffic to dedicated servers, ensuring that the remaining servers can handle all incoming traffic without any limitations.
Posted Mar 14, 2024 - 09:48 PDT
Monitoring
Our technical team has identified a significant increase in operations contributing to degraded performance. To mitigate the impact, we have implemented significant enhancements to facilitate faster operations. We are currently monitoring the results.
Our technical team anticipates that the backlog processing may require some time to complete. We will continue to monitor the environment, implement additional enhancements as necessary, and keep you informed of any further developments.
Posted Mar 13, 2024 - 22:53 PDT
Investigating
Incident Description:
We are currently experiencing substantial delays in processing inventory files uploaded to the IT Asset Management platform in the NA region. This delay is impacting multiple organizations, resulting in delayed updates to their inventory records.

Priority: P2

Restoration activity:
Our technical teams are actively involved in evaluating the situation. Additionally, we are exploring potential solutions to rectify the issue promptly.
Posted Mar 13, 2024 - 20:31 PDT
This incident affected: Flexera One - IT Asset Management - North America (IT Asset Management - US Inventory Upload).