The imgix URL API service failed to reliably service new render requests starting at 17:21 GMT / 09:21 PST on November 21, 2017. The root cause of the service outage had to do with a failure in our image fetching layer to sustain an increased delay in image fetches due to upstream network performance issues outside of our control. Our engineers were able to identify the core issue within a few minutes of its detection, but it took until 20:20 GMT / 12:20 PST to fully resolve the cascading consequences that the issue caused. From approximately 18:34 GMT / 10:34 PST until 20:20 GMT / 12:20 PST, we were in a recovering, but degraded state.
At the high point of the service interruption, new image renders and image revalidations were failing approximately 15-20% of the time for most customers. This did not impact images that were fresh in our cache and/or previously requested through the imgix CDN, which makes up over 90% of our typical request volume. Depending on the nature of specific use cases, some customers were more impacted than others. In particular, customers with lots of new content or content with short cache TTLs would have felt the most impact.
When a new image request is received, the imgix system must first either look up the image in our origin cache system, or failing that, fetch it from the customer origin. This process is called a “pre-flight” request. Additional complexity is incurred during this phase in order to facilitate optimal workload distribution and intelligent behavior based on the original object characteristics.
In this instance, upstream network performance issues began causing pre-flight requests to slow down considerably. Given that our customers utilize a variety of origins, located around the world, each with their own performance characteristics, our systems tend to retry requests aggressively and wait on longer-than-usual timeouts when an origin is found to be slower. While this behavior can help to smooth over issues we observe with some customer origins, it also exposes our system to greater risk of resource exhaustion. When all origins suddenly slow down, such as when there is network congestion that impacts all of imgix’s outbound traffic, we begin to build up a backlog of fetch requests waiting on images to load from the respective origins. This occurred slowly enough prior to the eventual service interruption that it did not immediately trigger our monitoring thresholds. Over time, we began to receive errors from our pre-flight requests going unanswered long enough to trigger downstream timeouts. This resulted in the original image request itself returning a 500-level response code rather than an image.
We received alerts starting at 17:21 GMT / 09:21 PST indicating that our resources were reaching a high utilization. At the same time, we started to get alerts for elevated error rates coming out of our fetch layer. Our on-call engineer responded by 17:22 GMT / 09:22 PST and raised the rest of the team to combat the issue. By 17:27 GMT / 09:27 PST, the scope of the issue had been narrowed down to the fetching layer. All evidence pointed at slow upstream performance issues in San Jose, which was corroborated by other cloud service providers who were experiencing similar issues at the same time.
Our first move was to deploy configuration changes to our pool of fetch nodes and take steps to reduce their individual workloads. This did not prove to be sufficient to mitigate the problem, and by 18:04 GMT / 10:04 PST, it was determined that the fastest path to get through our work backlog was to spin up additional, higher spec fetch nodes in order to minimize the time that requests spent in the queue. This helped us to absorb the slow origin load times.
It took longer than expected to coordinate the deployment of these systems. In particular, we did not have cold spare systems which we could easily activate. Our playbooks for deploying this type of service had become out-dated due to recent changes, and so that created some additional steps to work through to deploy new hosts. By 18:34 GMT / 10:34 PST, we had enough additional fetch nodes in production that we were processing 90-95% of incoming render requests successfully and began to observe reduced error rates. This behavior continued until 20:20 GMT / 12:20 PT when we observed that the service was completely restored to its original state.
Through our internal review process, imgix has decided to take the following courses of action in order to mitigate the impact or entirely eliminate future incidents of this nature.