Reports of certificate issues

Incident Report for imgix

Postmortem

Due to a mis-configuration at imgix’s CDN partner, imgix served a revoked (but otherwise valid) TLS certificate for imgix- and customer-provided hostnames used in conjunction with the imgix service for a period of time on Friday, Nov 16 2017 (UTC).

All client requests made to the imgix CDN during this period could have been impacted by this issue. The impact was reduced by three factors:

  1. Most common web browsers do not actively check for certificate revocation, and as a result end users may not be aware of any certificate trust issues associated with a revoked certificate.
  2. The revoked certificate was only served to browsers which made an SNI request, which is commonly supported but not universal.
  3. 10% of the imgix CDN servers were properly configured and not vulnerable to this issue.

Combined, these factors dramatically limited the observable impact to end users, although all imgix customers who make use of TLS in conjunction with the imgix CDN could potentially have been impacted.

The invalid certificate was revoked by imgix’s Certificate Authority around Friday, Nov 16 2017, 0500 UTC. The process to replace it with a valid certificate across the entire imgix CDN fleet began at Friday, Nov 16 2017, 1000 UTC and was completed by Friday, Nov 16 2017, 1130 UTC.

imgix actively monitors the performance and availability of the imgix CDN around the world, which includes checking for TLS certificate validity. However, this incident has exposed two critical gaps in our monitoring coverage:

  1. Our monitoring checks did not utilize SNI, and so could not have identified this incident. The correct certificate was always served to non-SNI traffic.
  2. imgix only examined certificate validity by looking at the begin / end valid dates, certificate common name and the certificate trust chain. It did not check for certificate revocation, and so could not have identified this incident.

imgix has corrected the underlying configuration issue which caused multiple versions of its certificates to be stored and served depending on SNI vs. non-SNI request patterns. Improvements to CDN monitoring will be implemented in the next several weeks in order to assist with the identification and resolution of this type of problem in the future, should it ever reoccur.

Posted Nov 17, 2017 - 17:34 PST

Resolved

This incident has been resolved.
Posted Nov 17, 2017 - 03:31 PST

Identified

We have identified the issue and we're working on releasing a fix.
Posted Nov 17, 2017 - 02:22 PST

Investigating

We're currently investigating reports regarding certificate issues. We'll update when we obtain more information.
Posted Nov 17, 2017 - 02:04 PST