What happened?

On June 13, 2020 at 19:02 UTC a faulty network configuration was deployed, causing network accessibility issues between some of the imgix servers. This triggered automated alerts, the issue was quickly identified, and the faulty configuration was removed. Service immediately began to recover by 19:18 and was completely restored by 19:23.

How were customers impacted?

During this time customers may have seen elevated errors for uncached derivative images as well as problems performing operations within the imgix Dashboard. Previously cached derivative images were not impacted.

What went wrong during the incident?

Standard procedures around networking changes to a relatively new component in the imgix infrastructure had not yet been fully socialized with all team members. This gap led to deviation from our ideal standard processes and resulted in the error.

What will imgix do to prevent this in the future?

We are already in the process of improving documentation and ensuring that everyone who has network access stays up-to-date on our internal procedures.