Timeframe

December 9th, 11:08 AM UTC - December 10th, 6:38 AM UTC

Impact

A small percentage of image requests served by a single machine in the EU region experienced quality degradation.

Root Cause

A hardware failure on one of the rendering machines resulted in images with artifacts being delivered to users.

Contributing Factors

An absence of automated testing for faulty renders across active machines caused assets to be cached undetected. The lapse between when the issue was detected and when the issue began culminated in an unknown number of renders that needed to be purged from the cache.

Along with inadequate cache tooling for purging affected images, these factors caused the issue to fully resolve in a longer timeframe than expected.

Corrective Actions

To prevent this issue from happening in the future, we will:

• Investigate and implement automated image quality testing for active rendering machines
• Replace and refresh machines on a more frequent basis to minimize impact of single machine failures
• Upgrade cache tooling for targeted issue resolution