What happened?

On December 12, 2021, 15:23 UTC, some imgix customers may have experienced issues with logging into their Dashboard and deploying Sources.

Login issues were resolved by 16:23 UTC. Source deployment issues were resolved by 17:43 UTC.

This incident only affected Web Administration tools - the rendering service was not affected.

How were customers impacted?

Customers attempting to log into the dashboard may have encountered a login error. Once the login issue was resolved, it was discovered that Source deploys were delayed.

By 17:43 UTC, all Source deployment issues had been fully resolved.

What went wrong during the incident?

Our team quickly discovered that a major cloud provider was experiencing a regional outage. This affected several web administration components, such as imgix Dashboard logins and delayed Source deployments.

Dashboard logins recovered on their own once the cloud provider recovered, though Source deployments remained delayed. It was assumed that Source deployments would eventually also resolve without intervention, though further investigation indicated that Source deployments required a manual fix, which took effect by 17:38 UTC.

What will imgix do to prevent this in the future?

We will be auditing multi-region contingency plans for all web administration components along with implementing more monitoring to ensure that impacted components can be identified sooner.