Description: IT Visibility - EU & NA - Data Processing was Delayed
Timeframe: May 5th, 07:00 AM PDT to May 6th, 6:38 AM PDT
Incident Summary:
On May 6th, at 2:32 AM PDT, technical staff observed a processing backlog in both NA and EU regions. As a result, customers may have experienced delays retrieving inventory data due to a processing backlog. EU data was up to 9 hours out of date and NA was up to 3 hours out of date.
Technical staff found that one of the consumer cluster data processing devices was unable to read data from the data streaming service. Additional subject matter experts were engaged from the other areas to assist with the investigation. Further investigation revealed that data encryption had been enabled on the streaming service and the impacted device did not have the correct roles assigned, which rendered its ability to decrypt and read data from the service.
Staff updated the device’s access roles in both EU and US at 3:35 AM PDT, following which health checks were completed successfully. At 5:23 AM PDT, the NA backlog processing was completed, and the EU backlog processing was completed at 6:38 AM PDT. After additional monitoring, technical staff declared the incident to be resolved.
Root Cause:
Infrastructure deployment across the EU and US environments altered some of the policies that enable the consumer storage cluster to decrypt the data from the streaming service.
Corrective Action:
• Technical staff updated the device’s access roles in both EU and US which authorized the device to decrypt and read data from the streaming service successfully
• Technical staff will work on enhancing coordination among teams during deployments to avoid service disruptions in the future