On April 13, 2023, between 17:09 UTC and 17:32 UTC, imgix experienced a partial outage affecting non-cached renders. During this time, requests to cached assets continued to serve a
200 response, while requests to non-cached assets returned a server error.
A fix was implemented at 17:32 UTC, restoring service.
Between 17:09 UTC and 17:32 UTC, requests to the Rendering API for non-cached renders returned a server error, with 9% of all requests to the Rendering API returning an error at the height of the incident.
We identified an error in one of our connections to customer origins. This error lead to significant slowdown in the retrieval process of new assets from customer origins. The errors rapidly grew in a short amount of time, causing our Rendering API to return 5xx errors.
To restore the service, our engineers redirected some of our network traffic. The service was fully restored by 17:32 UTC, but some errors persisted and were being served from the cache until they were completely cleared at 17:35 UTC.
We have taken the following steps to prevent this issue from re-occurring:
We are in the process of implementing the following: