Description: Flexera One – IT Asset Management – NA – Intermittent Errors Accessing Menu Items

Timeframe: September 26th, 8:07 AM to September 26th, 8:43 PM PDT

Incident Summary:

On Monday, September 26th, at 12:15 PM PDT, we received reports of customers experiencing errors and ITAM menu items not loading when the webpage was refreshed. The issue was reported to be sporadic, with reports from only a small number of customers.

Monitoring and health checks did not indicate any issues, prompting an investigation into the customer’s network. There were no errors observed on the customer’s network, however, technical staff found that no calls were being made by the application to obtain the JSON Web Token (JWT) which halted the end-to-end communication.

At 5:00 PM PDT, technical staff found the incident was related to the Flexera UI change that was performed earlier that day. As a part of the change, the Persist-user settings were introduced in Flexera One, allowing users to retrieve the current org by using the saved preferences.

At 7:31 PM PDT, technical staff identified communication delays between the backend service and a webhook that triggered a secure message to the application to call the JWT endpoint and retrieve the menu items.

After further analysis, a code change was tested in the lower environment at 8:36 PM PDT. At 8:43 PM PDT, the change was successfully deployed to Production. The impacted customers also confirmed that they could access ITAM and retrieve menu items successfully, following which the incident was declared resolved.

Root Cause:

A root cause was identified to be communication delays between the backend service and a webhook that triggered a secure message to the application to call the JWT endpoint and retrieve the menu items.

Contributing Cause:

Following the deployment, technical staff performed additional tests, but the existing unit tests did not detect the issue as each function did its role and passed correctly. In addition, due to the intermittent nature of the issue, no immediate failures were observed during the health checks.

Corrective Action:

• A code change was introduced in the environment that would trigger the secure message to the application to call the JWT endpoint every time the org is changed, and the webpage is refreshed in the application
• Technical staff will enhance testing procedure during deployments and add end-to-end smoke tests that run in the real environment/domain to check the JWT token requests