At June 20th 2019 15:11 UTC, the imgix service saw elevated origin latency when retrieving images. This caused an increase in latency and error rates for newly rendered derivative images. This lasted for approximately two hours with the issue subsiding by 17:20 UTC.
There were sustained error rates for two to three percent of all traffic for approximately two hours. While many requests were being served properly, there was latency on new renders. Note that previously rendered and cached derivative images were not affected.
Our service monitoring identified the elevated error rates and we began remediation efforts immediately. While we did see some positive benefit from the initial mitigation, it was not as complete as we had hoped for. Subsequent efforts did enable us to successfully resolve the issue.
We have identified work which can be done to aid in fault isolation when encountering increased latency while requesting content from origins. Several of these changes have been put in place already and work is progressing on others. Additional work is scheduled to provide advance warning when origin latency increases in order to better isolate faults and deploy mitigating measures.