Delay in purge requests
Incident Report for imgix

What happened?

On October 26, 7:52 UTC we received a purge request containing unparsable data, which caused elevated delays in fulfilling new purge requests. Our engineering team identified the issue, implementing a short term fix to block corrupted purge requests. The purging service completely recovered at 9:48 UTC.

How were customers impacted?

Customers may have seen up to a 1.5 hour delay in purge requests, though no customers reported any issues related to the incident.

What went wrong during the incident?

During this incident, we found that our internal logs lacked some of the proper data to isolate the issue to the purger. This slowed our initial investigation, though once the issue was identified, our engineers pushed a fix to immediately resume purging operations.

What will imgix do to prevent this in the future?

We will be implementing even more stringent input validation of purge requests to prevent re-occurrence of this issue. We will also be updating our troubleshooting processes to speed up root cause analysis of purging issues in the future.

Posted Oct 29, 2020 - 11:57 PDT

This incident has been resolved.
Posted Oct 26, 2020 - 02:46 PDT
This incident has been resolved. Purging response times are back to normal levels.
Posted Oct 26, 2020 - 02:41 PDT
The issue has been identified and a fix is being implemented.
Posted Oct 26, 2020 - 02:37 PDT
Our engineering team is investigating a delay in purge requests. The rendering service is not impacted.
Posted Oct 26, 2020 - 01:56 PDT
This incident affected: Purging.