On November 17, 00:15 UTC the imgix rendering service was affected by packet loss stemming from one of our network providers.
During brief periods between the hours of 8:15 UTC and 10:40 UTC , a small percentage of requests (1.7%) returned the error message 503 No Healthy Backends
. These periods lasted between one and three minutes and reoccurred several times. The incident was completely resolved by 10:40 UTC.
We began experiencing packet loss stemming from one of our network providers which caused some images to return a 503
response code. The transient and limited impact of the incident stalled our escalation processes and obfuscated our decision tree for remediating incidents. This also prevented the status page from being updated since the issues would disappear as quickly as they had started.
We are redefining escalation conditions in regards to recurring, self-solving incidents. We are also updating our tooling to both implement better monitoring on transient issues and to provide resilience when experiencing packet loss between transit providers.