Description: Cloud Management Platform (CMP) - NA - Self-Service - Shard 4 - Potential Errors Launching New Cloud Apps with Plugins

Timeframe:  9 September 2024, 10:30 AM PDT to 9 September 2024, 12:24 PM PDT

Incident Summary

‌

On Monday, 9th September 2024, 10:30 AM PDT, following a scheduled weekend maintenance release, our team identified an issue impacting customers using the Cloud Management Platform (CMP) on Self-Service Shard 4. Customers attempting to launch new Cloud Apps that utilized Plugins encountered errors. This issue was isolated to Shard 4, and was caused by a feature update deployed during the maintenance window.

To prevent further disruptions, an emergency rollback was initiated immediately after the issue was identified. During the rollback process, some customers experienced brief slowness and delays in launching Cloud Apps and performing other operations. The rollback was successfully completed by 12:24 PM PDT on 9 September 2024, restoring full functionality to Shard 4.

‌

Root Cause

The issue occurred due to an update applied during the scheduled maintenance window. This update introduced a feature that affected the functionality of Plugins when customers attempted to launch new Cloud Apps. The change has been rolled back and the erroneous code has been reviewed and fixed.

‌

Remediation Actions 

‌

·        Emergency Rollback: Upon identification of the issue, the team immediately implemented a rollback of the changes deployed during the scheduled maintenance.

·        Restoration of Services: After the rollback was completed, normal operations on Shard 4 were restored, and customers could launch Cloud Apps with Plugins without encountering errors.

‌

Future Preventative Measures 

‌

 Pre-release Testing Improvements:

• Enhance testing protocols for systems operating on legacy platforms, ensuring new features are tested for compatibility before deployment.
• Expand regression testing to cover more scenarios specific to Shard 4.

Compatibility Checks:

• Implement a compatibility validation process as part of the deployment. This will ensure that any new features or updates are validated against all shards before release.

Monitoring Enhancements:

• Increase monitoring capabilities, specifically focusing on plugin functionality and key operations such as Cloud App launches. This will enable faster detection of issues, allowing the team to respond before customers are significantly impacted.