Postmortem

What happened?

On May 23, 2024, at 19:23 UTC, an increased load on the rendering infrastructure was detected. Actions were taken to scale out our system to handle the additional traffic. This incident was resolved at 19:36. 

How were customers impacted?

During the incident, customers experienced increased error rates for recent renders, intermediate errors increased in our system, and response times for requests increased. 

What went wrong during the incident?

During the incident, our team implemented a service change that led to assets being dropped. This led to an increase in requests to our system. The increased requests to our system led to 429 and 5XX errors.

What will imgix do to prevent this in the future?

To prevent similar incidents, we will:

• Improve procedures for pre-scaling instances during critical updates.
• Conduct impact assessments before issuing significant changes.
• Enhance monitoring and alerting systems to predict and manage load increases better.