This article contains quite a bit of inaccurate information.
Despite the size of 22 double-XL instances, they were a small portion of our overall footprint in EC2; it takes well over that kind of capacity to run the platform.
Those instance types all happened to be in one availability zone for a variety of reasons. Our platform overall does not live in a single zone.
Losing machines is not a problem for us (we cycle them constantly, in fact). Normally losing even that many machines would not even be noticed by our customers; this was an unusual case in which several factors cascaded into a larger problem.
To be clear, this downtime (45 minutes or so, with full normal state by 90 minutes) was, unfortunately, our fault - not Amazon's. EC2 instances vaporizing is an expected part of using the service.
We've made a couple of operational changes that will prevent these issues in the future, and we sincerely apologize to any customers who were affected.
Despite the size of 22 double-XL instances, they were a small portion of our overall footprint in EC2; it takes well over that kind of capacity to run the platform.
Those instance types all happened to be in one availability zone for a variety of reasons. Our platform overall does not live in a single zone.
Losing machines is not a problem for us (we cycle them constantly, in fact). Normally losing even that many machines would not even be noticed by our customers; this was an unusual case in which several factors cascaded into a larger problem.
To be clear, this downtime (45 minutes or so, with full normal state by 90 minutes) was, unfortunately, our fault - not Amazon's. EC2 instances vaporizing is an expected part of using the service.
We've made a couple of operational changes that will prevent these issues in the future, and we sincerely apologize to any customers who were affected.