At 6:34pm UTC on 2016/7/26, we were notified of degraded rendering performance and elevated 504 status code responses. By 6:56pm UTC, we had started to diagnose the cause of this issue.

An ad-hoc query to our load balancers was run, which caused a table lock. This backed up the fetchers and caused their backlog to bloat. Although the backlog should run completely in the background, this revealed a bug that backlog consumption to block live requests. This appeared to users as render slowdowns and timeouts. We stopped the fetcher, wiped the cache, and restarted.

By 7:13pm UTC, we had resolved the error rate and most customers had working images. By 7:20pm UTC, the user issue had been resolved, we’d identified the root cause of the issue and had begun resolving it. By 8:15pm UTC, we pushed out a fix for the identified bug and fully resolved the issue.