Postmortem

What happened?

On Feb 20, 2024, at 19:00 UTC, Uploads using the `/api/v1/sources/upload/` API endpoint experienced slowness and some timeouts during this time.  

The issue had been completely resolved by Feb 21, 2024, 20:00 UTC.

How were customers impacted?

Between Feb 20, 2024, 19:00 UTC, and Feb 21, 2024, 20:00 UTC, some requests to our upload API experienced slow responses, and a subset of requests resulted in timeout errors.

Uploads using the `/api/v1/sources//upload-sessions/` endpoints were unaffected; uploads using the Asset Manager UI were unaffected.

What went wrong during the incident?

Two compounding issues caused the slowdown. The first is a service update creating an issue with our /api/v1/sources/upload/` API endpoint.  Secondly, a simultaneous and separate slowdown affected the cloud servers responsible for executing the upload actions. 

The service update issue increased the load on our upload function, which was already under strain due to the cloud servers experiencing a slowdown. These factors combined to cause the slow and timed-out responses.

What will imgix do to prevent this in the future?

We have streamlined the upload process so that this cannot happen again.