At 5:00pm UTC on 2016/7/4, we were notified of degraded rendering performance and errors in image fetching from customer origins. By 5:22pm UTC, we had started to diagnose the cause of this issue.

When files are first requested from the origin, a primary database connects to the origin, requests the files, and upon success, caches the file temporarily before flushing the file to a centralized origin cache database. This primary database was misconfigured and used an incorrectly-sized partition for logs and data directories - the partition filled up quickly and cause the primary database to be unwritable.

Once we were aware of the issue, we temporarily stopped fetcher processes, wiped their cached, and restarted. This change restored traffic immediately. By 5:40pm UTC, we had resolved the error rate and most customers had working images being rendered. By 6:10pm UTC, we’d identified the root cause of the issue and had begun resolving it. We shut down the primary database and synced logs and data to the proper partition. By 7:29pm UTC, we’d fully resolved the issue.

As a preventative measure, we tightened our alerting to catch these issues more ahead of time and solved the underlying issue.