Description: Saas Manager - APAC - Service degradation affecting access to SaaS manager service
Timeframe:
Incident Summary
On Tuesday, September 16, 2025, at 9:06 PM PDT, we identified an issue impacting the SaaS Manager in the APAC region. Consequently, all customers in APAC were unable to access the application, while services in the EU and NAM regions continued to function normally.
Affected customers attempting to access SaaS Manager or All SaaS Licenses encountered the following error message: “Retrieving SaaS Licenses failed. Please contact support.” Similarly, users trying to access Managed SaaS Applications received the error: “An unexpected error occurred while loading managed applications.”
Initial investigations revealed that the problem stemmed from a third-party service being halted, which led to the failure of related health checks. To restore services, our teams disabled the health checks, successfully resolving the issue by 10:57 PM PDT on September 16. We continued to monitor the platform closely for stability and to prevent any recurrence of the issue. Following extensive monitoring, we confirmed that the issue was fully resolved on September 17, 2025, at 4:05 AM PDT.
However, on September 17, 2025, at 5:24 PM PDT, a recurrence of the outage occurred in the APAC region. The issue was triggered when the monitoring service for the third-party analytics service was brought down, once again impacting the service instances and causing them to crash. This resulted in another outage until September 17, 2025, at 9:28 PM PDT, when corrective actions were applied.
Root Cause
Initial investigations confirmed that a third-party analytics service in the APAC region had been intentionally stopped since it was meant to be deprecated. This action caused the service responsible for API routing to become non-functional. Two dependent services were identified as unresponsive, which resulted in instance crashes and unavailability of SaaS Manager functionality for all APAC customers.
In the second incident, the monitoring service for the third-party platform was brought down, which again triggered dependency failures and caused another outage.
Remediation Actions
· Disabled the health check for the third-party analytics service to prevent cascading failures.
· Stabilized the environment, allowing instances to run successfully and pages to load as expected.
· Validated restored functionality across multiple customers.
· Monitored the environment continuously following the workaround.
· Fully restored access to SaaS Manager and Managed SaaS Applications for all APAC customers on September 17, 2025, at 4:05 AM PDT.
· Following the recurrence, we removed all code related to this service from our product across all production environments (APAC, EU, and NAM) since only the health check for the third-party service was disabled during the first instance, which restored service at that time.
· Conducted extended monitoring across APAC services to validate the fix.
· Fully restored access to SaaS Manager for all affected APAC customers on September 17, 2025, at 9:28 PM PDT.
Future Preventative Measures
· Review and document dependencies on the third-party analytics service across all regions before making further changes.
· Initially, we only disabled the health check for this service. However, as a permanent solution, we removed all code related to this service from our product to ensure this issue does not recur.
· Introduce additional safeguards to ensure service dependencies are evaluated and tested before disabling or removing critical components.
· Continue extended monitoring across all SaaS Manager environments to ensure stability and resilience.