Issues communicating with Amazon S3

Incident Report for imgix

Resolved

Our temporary fix is still in place and continues to provide an adequate solution to the problem. As such, we will mark this issue as resolved and remove our temporary fix at a date in the near future (likely 2 weeks). Performance for customers using S3 will be unaffected during this time.
Posted Nov 19, 2015 - 16:47 PST

Update

Traffic levels and error rates have stabilized with our temporary solution. We will look to safely remove our temporary fix within the next 24 hours, if possible. Next update by November 20, 01:00:00 UTC.
Posted Nov 18, 2015 - 17:44 PST

Monitoring

We've implemented a temporary solution at that has alleviated the issue. Error rates are returning to normal. Next update by 01:45:00 UTC.
Posted Nov 18, 2015 - 17:23 PST

Update

To provide a temporary fix, we are attempting to ignore the Amazon IP addresses that produce the packet loss. We will provide further updates by 01:30:00 UTC.
Posted Nov 18, 2015 - 17:13 PST

Identified

We've identified the issue lies with AWS and not a backbone provider.

The incident is currently not present on the AWS Service Health page: http://status.aws.amazon.com/
Posted Nov 18, 2015 - 16:47 PST

Update

The current issues are resulting in 50-70% packet loss isolated to the following IP addresses in the route of problematic requests:

205.251.229.145
54.240.242.108
54.240.242.8
54.240.242.148
54.240.242.128

According to arin.net, these IP addresses are under the control of Amazon.
Posted Nov 18, 2015 - 16:43 PST

Investigating

Our team has been alerted to issues stemming from an inability to communicate with Amazon S3. Early signals indicate this may be an issue with a backbone provider. We are actively investigating and will report updates as they become available. Next update by 01:00:00 UTC.
Posted Nov 18, 2015 - 16:36 PST