Image Manager is not displaying any images
Incident Report for imgix

What happened?

On August 19, 2021, 15:45 UTC images had disappeared in customer instances of the Image Manager. This incident was marked fully resolved about an hour later at 16:33 UTC.

After the initial incident had been resolved and images had been restored, the issue resurfaced 4 days later on August 23, 2021, 16:24 UTC. During the second incident, the first attempt at recovery had duplicated images in the Image Manager. This incident lasted for a very brief moment and was resolved 8 minutes later at 16:32 UTC. 

How were customers impacted?

In the first incident, images that previously had shown in our Image Manager had disappeared. If users opened up the Image Manager user interface or had called Image Manager's list assets endpoint, they would not find any images.

In the second incident, images that previously had shown in our Image Manager had duplicated themselves. The end result is that there were 2 copies of every image in the Image Manager.

Since imgix does not host images, this issue only affected interactions with the user interface and the Image Manager API. No data or images were lost during this incident; it was a display-only issue. Origin images continued to be stored at customer origins.

What went wrong during the incident?

On August 19, 15:45 UTC, imgix identified that at least some customers were seeing their Image Manager without any images. The issue was escalated, and our team began investigating the cause. Eventually, the issue was traced to a bad image index that had cleared out the Image Manager state for customers.

After identifying the problem, our engineers recovered a previous image index, restoring images to the customer Image Manager instances.

On August 23, 16:24 UTC, the incident resurfaced. The same fix was applied using the same tooling, though due to improper configurations, it had caused every image in the Image Manager to be duplicated. After re-configuring the tooling, it was executed again, restoring the Image Manager for customers.

What will imgix do to prevent this in the future?

We will be developing tooling to monitor the health of the Image Manager, along with improving internal documentation regarding Image Manager remediation. We will also be tuning image indexing to eliminate and handle the conditions that had caused image indexes to function incorrectly.

Posted Aug 30, 2021 - 09:20 PDT

The incident affecting the Image Manager has been resolved.
Posted Aug 19, 2021 - 09:33 PDT
A fix has been implemented and we are monitoring the results.
Posted Aug 19, 2021 - 09:27 PDT
We are investigating an issue with images not being displayed in the Image Manager.

The rendering service is fully operational and is not affected by this incident.
Posted Aug 19, 2021 - 09:05 PDT
This incident affected: Web Administration Tools.