ITAM - APAC - IT Visibility Data Sync may not remove all stale data

Description: Flexera One - IT Visibility and IT Asset Management Integration Failure Leading to Stale Data

Timeframe: November 22nd, 7:19 PM to November 23rd 2:49 AM PST

Incident Summary

On Wednesday, November 22nd, at 7:19 PM PST, we received notification of an issue affecting customers utilizing the IT Asset Management and IT Visibility platforms in the APAC region. The problem originated from the integration of IT Visibility with IT Asset Management, leading to the retention of outdated inventory data in the system. Consequently, users of both platforms may have observed stale inventory information in their IT Visibility reports.

Our technical teams promptly initiated efforts to isolate the issue and develop a strategy for expediting the recovery process. Simultaneously, they compiled a list of devices for affected organizations, conducted manual tests to verify the connection between IT Asset Management and IT Visibility's front service API, and thoroughly inspected the code.

Upon conducting a comprehensive investigation, on Thursday, November 23rd, at 12:16 AM PST, our teams discovered that the integration failure stemmed from the absence of configuration for the IT Visibility front service URL in IT Asset Management. Further examination revealed that, while performing certain standard configuration procedures, some default settings were unintentionally retained.

At 12:47 AM PST, our technical staff promptly addressed the URL configuration issue. Subsequently, they monitored application logs to verify the resolution. At 2:27 AM PST, our teams identified that certain requests were not processed successfully, indicating a misconfiguration of the public key for the cloud service. This issue was rectified by 2:44 AM PST.

Following health checks and log inspections, the technical teams confirmed the successful restoration of IT Visibility integration with IT Asset Management. Data synchronization was validated to be operating as intended, leading to the resolution of the incident at 2:49 AM PST.

Root Cause

Primary Root Cause: The integration failure stemmed from the absence of configuration for the IT Visibility front service URL in IT Asset Management. Further examination revealed that, while performing certain standard configuration procedures, some default settings were unintentionally retained.

Contributing Factor: The integration failure was further complicated by a misconfiguration in the public key for the cloud service.

Remediation Actions

1. Isolation and Strategy Development: Technical teams promptly isolated the issue and devised a strategy for expedited recovery.
2. Device List Compilation and Manual Tests: Simultaneously, teams compiled a device list for affected organizations and conducted manual tests to verify IT Asset Management and IT Visibility's front service API connection.
3. Addressing URL Configuration Issue: At 12:47 AM PST, technical staff promptly resolved the URL configuration issue identified as a key factor in the integration failure.
4. Resolution of Public Key Misconfiguration: At 2:27 AM PST, teams identified a misconfiguration of the public key for the cloud service and rectified it by 2:44 AM PST.
5. Validation of Data Synchronization: Following issue resolution, health checks and log inspections confirmed the successful restoration of IT Visibility integration with IT Asset Management, validating proper data synchronization at 2:49 AM PST.

Future Preventative Measure(s)

1. Robust Configuration Testing: Strengthen testing procedures for configuration changes, ensuring thorough checks before, during, and after scheduled tasks.
2. Continuous Monitoring: Implement continuous monitoring of key system parameters, including configurations, to promptly detect and address any anomalies before they lead to integration failures.