On October 23, 2023, between 21:43 UTC and 23:14 UTC, imgix experienced a partial outage affecting images served from the Rendering API. During this time, a small percentage (<0.45% on average) of non-cached requests returned a server error.
A fix was implemented at 23:02 UTC, which allowed the service to recover by 23:14 UTC fully.
Between 21:46 UTC and 23:14 UTC, requests to the Rendering API returned a server error, with 0.65% of all requests to our CDN returning an error at the height of the incident.
Additionally, Sources returned an unknown status between 21:06 UTC to 21:09 UTC. During this period, customers reported being unable to create Sources.
Our Rendering API experienced an unexpected interaction that caused a dramatic increase in server load. This caused error rates to increase as the network became overloaded slowly. The errors fluctuated between 0.07% to 0.65% until we resolved the issue.
To restore the service, our engineers re-configured our network traffic to handle the unexpected Rendering behavior.
During the incident, a separate issue (unrelated to rendering) impacted our Source data. This led to a delay in investigating the cause of the rendering errors.
We have taken the following steps to prevent this issue from recurring:
We are in the process of implementing the following: