SSL certificate errors
Incident Report for imgix
Postmortem

What Happened?

At approximately 19:19 UTC in June 26 2018, the imgix engineering team was alerted to issues regarding HTTPS connectivity to imgix customer sources. The team immediately began an investigation, which revealed that an imgix SSL certificate was marked as revoked by our issuing certificate authority.

At 19:36 UTC the imgix team determined that a previously in-service SSL certificate (which had subsequently been invalidated) was being served to clients of the imgix CDN. Further investigation revealed that our CDN partner re-deployed this invalid certificate during a routine production deployment due to human error.

In conjunction with our certificate authority and CDN partner, imgix began serving a valid certificate globally by 19:58 UTC. This resolved all known connectivity issues pertaining to customer HTTPS traffic. The imgix team continued to monitor the situation and marked the incident as fully resolved at 20:19 UTC.

How were customers impacted?

Connections to the imgix CDN utilizing TLS were not trusted by many major browsers during the incident period.

This is likely to have caused one or more of the following conditions:

  • Previously rendered image content failed to load.
  • Newly requested image content failed to render.
  • Customer web pages were marked as insecure.

This impacted all imgix customers (who utilize HTTPS) and their end users (who utilize impacted browsers) over an approximately 40 minute period.

What went wrong during the incident?

  • imgix team members had issues replicating the problem on certain browsers which contributed to a delay in identifying the root cause.
  • Communication channels between imgix engineering and some partners were not completely documented for the entire team.

What will Imgix do to prevent this in the future?

Through our internal review process, imgix has identified the following actions to mitigate or entirely eliminate the impact of future incidents of this nature.

  • Review certificate deployment procedures with the imgix engineering team and our CDN partner.
  • Identify additional steps for our CDN partner to take to reduce or eliminate manual intervention steps during the certificate deployment process.
  • Further refine our continuous monitoring of certificate health.
Posted Jun 26, 2018 - 15:56 PDT

Resolved
Service has returned to normal and certificate error issues are now resolved.
Posted Jun 26, 2018 - 13:19 PDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Jun 26, 2018 - 13:03 PDT
Investigating
We're currently investigating reports of SSL certificate issues. We'll update when we obtain more information.
Posted Jun 26, 2018 - 12:36 PDT