At 8:55pm UTC on 2016/7/7, we were notified of degraded rendering performance and errors in image fetching from customer origins. By 9:06pm UTC, we had started to diagnose the cause of this issue.

Content from origin is fetched on the first request, served, and cached in our origin cache through a background operation. The fetcher absorbs latency and any other irregularities while persisting the content in our cache using an internal work queue that shields slowdowns. A sudden backlog was observed that didn't catch up, which caused the timeouts that a very small percent of our traffic observed.

By 9:13pm UTC, we had resolved the timeout issue. We cleared the fetcher backlog and fixed the underlying issue with handling content. By 10:45pm UTC, we’d fully resolved the issue.