Description: Flexera One – IT Visibility – NA - Errors Downloading API Exports

Timeframe: October 5th, 2022, 1:00 AM to October 5th, 2022, 2:20 AM PDT

Incident Summary:

On October 5th, at 1:00 AM PDT, Following the Flexera One - IT Visibility API scheduled maintenance in North America, technical staff found that Flexera One IT Visibility Dashboards were not getting refreshed and API exports were unavailable. As a result, some customers may have experienced errors downloading exports or seeing the latest data on Flexera One IT Visibility Dashboard. ServiceNow exports were unaffected by this incident.

Scheduled maintenance commenced at 11:00 AM PDT. During the maintenance, staff encountered various issues that lasted beyond the maintenance window. As a part of the change, a new network load balancer (with new IPs/DNS zone) was introduced in the environment, but the API gateway was still pointing to the previously deployed network load balancer configuration for IT visibility services.

Staff also found that the existing policies were not synching during the deployment and the service provider configuration was also altered resulting in the orphaning of resources and policies.

The technical staff updated the API gateway configuration and redirected it to the new Network Load Balancer services, after which the load balancer was redeployed. At 2:10 AM PDT, technical staff reconfigured the service and manually re-attached the policies to various resources. At 2:12 PM PDT, orphaned resources were cleaned from the environment.

At 2:20 AM PDT, after additional health checks and monitoring, the incident was declared resolved.

Root Cause:

As a part of the change, a new network load balancer (with new IPs/DNS zone) was introduced in the environment. Analysis showed that the API gateway was still pointing to the previously deployed network load balancer configuration for IT visibility services resulting in subsequent issues

Contributing Cause:

Existing policies were not synching during the deployment and the service provider configuration was also altered resulting in the orphaning of resources and policies

Corrective Actions:

• The technical staff updated the API gateway configuration and redirected it to the new Network Load Balancer services, after which the load balancer was redeployed
• Reconfigured the service and manually re-attached the policies to various resources