Network issues affecting new renders
Incident Report for imgix

What happened?

On June 13, 2020 at 19:02 UTC a faulty network configuration was deployed, causing network accessibility issues between some of the imgix servers. This triggered automated alerts, the issue was quickly identified, and the faulty configuration was removed. Service immediately began to recover by 19:18 and was completely restored by 19:23.

How were customers impacted?

During this time customers may have seen elevated errors for uncached derivative images as well as problems performing operations within the imgix Dashboard. Previously cached derivative images were not impacted.

What went wrong during the incident?

Standard procedures around networking changes to a relatively new component in the imgix infrastructure had not yet been fully socialized with all team members. This gap led to deviation from our ideal standard processes and resulted in the error.

What will imgix do to prevent this in the future?

We are already in the process of improving documentation and ensuring that everyone who has network access stays up-to-date on our internal procedures.

Posted Jun 21, 2020 - 21:27 PDT

This incident has been resolved.
Posted Jun 13, 2020 - 12:45 PDT
Our engineering team has deployed some updates, restoring the service. We are currently monitoring the situation.
Posted Jun 13, 2020 - 12:30 PDT
We are currently investigating a network issue affecting newly rendered images and the dashboard. We will update once when we obtain more information.

Previously cached derivatives are not impacted.
Posted Jun 13, 2020 - 12:16 PDT
This incident affected: Rendering Infrastructure and Web Administration Tools.