On August 19, 2021, 15:45 UTC images had disappeared in customer instances of the Image Manager. This incident was marked fully resolved about an hour later at 16:33 UTC.
After the initial incident had been resolved and images had been restored, the issue resurfaced 4 days later on August 23, 2021, 16:24 UTC. During the second incident, the first attempt at recovery had duplicated images in the Image Manager. This incident lasted for a very brief moment and was resolved 8 minutes later at 16:32 UTC.
In the first incident, images that previously had shown in our Image Manager had disappeared. If users opened up the Image Manager user interface or had called Image Manager's list assets endpoint, they would not find any images.
In the second incident, images that previously had shown in our Image Manager had duplicated themselves. The end result is that there were 2 copies of every image in the Image Manager.
Since imgix does not host images, this issue only affected interactions with the user interface and the Image Manager API. No data or images were lost during this incident; it was a display-only issue. Origin images continued to be stored at customer origins.
On August 19, 15:45 UTC, imgix identified that at least some customers were seeing their Image Manager without any images. The issue was escalated, and our team began investigating the cause. Eventually, the issue was traced to a bad image index that had cleared out the Image Manager state for customers.
After identifying the problem, our engineers recovered a previous image index, restoring images to the customer Image Manager instances.
On August 23, 16:24 UTC, the incident resurfaced. The same fix was applied using the same tooling, though due to improper configurations, it had caused every image in the Image Manager to be duplicated. After re-configuring the tooling, it was executed again, restoring the Image Manager for customers.
We will be developing tooling to monitor the health of the Image Manager, along with improving internal documentation regarding Image Manager remediation. We will also be tuning image indexing to eliminate and handle the conditions that had caused image indexes to function incorrectly.