Description: Spot Ocean for AWS – Instance Provisioning Failures

Timeframe: September 24, 2026, 12:00 PM PDT to September 24, 2026, 2:54 PM PDT

‌

Incident Summary

‌

On September 24, 2026, Spot Ocean for AWS experienced an issue that prevented some customers from launching new instances. As a result, affected clusters were unable to scale as expected when additional compute capacity was required.

Existing instances and workloads remained operational throughout the incident. The impact was limited to new instance provisioning and cluster scale-up operations.

The technical team identified that a recent software change introduced unexpected behavior during the instance provisioning process. The change was rolled back, which cleared the provisioning errors and restored normal service operation.

Following the rollback, the underlying software issue was identified and corrected. The updated functionality has undergone additional validation to ensure the issue is fully addressed before it is reintroduced into the production environment.

Technical teams validated successful instance launches following the rollback, and customer validation confirmed that new instance provisioning was functioning as expected. The incident was resolved at 2:54 PM PDT on September 24, 2026.

‌

Root Cause

‌

The incident was caused by a recent software change that introduced unintended behavior during AWS instance provisioning. The change affected how certain cluster configuration information was processed, resulting in provisioning failures for some scale-up operations. While the change was designed to support tagging for specific AWS resources, its behavior also affected other supported cluster configurations. Under certain conditions, the updated tagging logic resulted in duplicate tag information being submitted as part of the instance provisioning request. AWS rejected these requests because the tag information did not meet the expected requirements. As a result, Spot Ocean was unable to successfully complete the provisioning of some new instances.

This primarily affected new instance launches and cluster scale-up operations. Existing instances had already been successfully provisioned and therefore continued to operate normally throughout the incident.

The underlying functionality has since been corrected, and additional validation is being performed to ensure the updated behavior works consistently across the supported AWS cluster configurations before it is returned to production.

‌

‌

Remediation Actions

Our teams performed the below actions to restore the services to normalcy:

• Rolled back the software change, restoring normal instance provisioning.
• Identified and corrected the underlying issue in the affected functionality.
• Performed additional validation of the corrected functionality.
• Introduced additional safeguards to allow the updated functionality to be rolled out in a controlled manner.
• Validated that new instance provisioning and cluster scaling operations were functioning as expected following the rollback.

‌

Future Preventative Measures

• Expand automated testing for instance provisioning and cluster scaling scenarios.
• Increase regression testing across supported AWS cluster configurations.
• Expand end-to-end testing for scaling workflows.
• Improve monitoring and alerting for provisioning and scaling failures.
• Strengthen validation of changes before production deployment.
• Introduce additional safeguards for controlled rollout of changes that affect provisioning behavior.
• Increase testing coverage for edge cases and error-handling scenarios.