On April 26, 2021 21:00 UTC, the imgix service experienced elevated latency when retrieving images via our origin cache. This caused an increase in error rates for some uncached derivative images. Our engineering team implemented a remediation that had error rates subsiding by 21:28 UTC and with full restoration of the service by 21:45 UTC.
During the incident, customers may have noticed some uncached derivative images return an error. Up to 10% of requests failed to return successfully.
Cached derivative images were not impacted and continued to be served as normal.
Our engineers were alerted to an increased amount of failed origin requests. The issue was quickly identified as traffic which was outside of our terms of service. The traffic was isolated while we worked with affected parties for a longer term solution.
Response time of our team was increased given ongoing projects regarding suspicious traffic. However, we are still seeing that there are improvements that can be made so that process response times will have a faster turnaround time. Moving forward, we will continue to evaluate how to further optimize our service to prevent situations like this.