On May 23, 2024, at 19:23 UTC, an increased load on the rendering infrastructure was detected. Actions were taken to scale out our system to handle the additional traffic. This incident was resolved at 19:36.
During the incident, customers experienced increased error rates for recent renders, intermediate errors increased in our system, and response times for requests increased.
During the incident, our team implemented a service change that led to assets being dropped. This led to an increase in requests to our system. The increased requests to our system led to 429
and 5XX
errors.
To prevent similar incidents, we will: