Render slowdowns and timeouts.

Incident Report for imgix

Postmortem

At 8:42pm UTC on 2016/7/17, we were notified of degraded rendering performance and elevated timeout errors.

Our load balancers advertises availability on each of its nodes separately. We found an issue where a lack of availability was advertised, causing workers to be overloaded. This resulted in the degraded performance and timeouts that users reported.

By 9:31pm UTC, we had resolved the error rate and most customers had working images. We restarted our load balancing processes and identified the issue. By 10:22pm UTC, we’d fully resolved the issue.

Posted Aug 30, 2016 - 11:03 PDT

Resolved

This incident has been resolved.

Posted Jul 17, 2016 - 15:35 PDT

Monitoring

We have identified an issue in our load balancing that was causing a cache rebalance, leading to timeouts due to renders queueing for too long. A fix is in place and service should be behaving normally. Post-mortem to follow. We will continue monitoring.

Posted Jul 17, 2016 - 14:31 PDT

Investigating

Experiencing image timeouts and slow renders. Team is on it. Will follow up shortly.

Posted Jul 17, 2016 - 13:42 PDT