> For these reasons, and our strong focus on uptime, we abandoned EBS entirely, starting about six months ago, at some considerable cost in operational complexity (mostly around how we do backups and restores). So far, it has been absolutely worth it in terms of observed external uptime.
So what do you use now for your persistent storage? This might be the most interesting part.
All our persistent storage is on "ephemeral" drives. If we lose the instance, we lose the data, so we have a lot of redundant slaves and backups (see my other comment).
This is a good approach if you can get enough redundancy with your instances and if instance failure is uncorrelated.
However, what do you do if there's a massive outage that affects most or all of your instances simultaneously? Diversification that fails when you need it the most isn't very helpful.
As I mention in the post, we go across AWS availability zones: these systems have physically separate power and networking. When they fail together, it's because they are connected at the software level -- usually EBS. If you're not using EBS, then Amazon's multi-AZ pattern is sufficient: just make sure your instances are spread across AZs.
Of course, east-1 has fallen off the map entirely on at least one occasion -- for that reason we keep a "seed" set of warm databases in us-west; if east-1 were to have an extended outage we have a disaster recovery plan that involves transferring DNS to IPs in west and spinning up new app instances there.
How do you manage this? In my experience ephemeral storage is about 8GB. Are you able to just parition all data so that you never reach this limit, or is there a way to increase ephemeral space that I don't know about?
Even the smallest instances have 160GB of storage, all the way up to 3TB on the cc2.8xlarge. New high I/O instances have 2TB of "holy shit fast, destroys my $1000 enterprise SSD" storage. (just tested it last week)
New high I/O instances have 2TB of "holy shit fast, destroys my $1000 enterprise SSD" storage.
Just a tongue-in-cheek comment: Those instances also cost over $2000 USD per month. For the same money you could buy a new Dell Server with 2-4 SSDs, every month. ;)
Each EC2 instance type has a different amount of ephemeral storage, from 160 GB to 3.3 TB. You need to arrange for it to be mounted when you launch the instance.
So what do you use now for your persistent storage? This might be the most interesting part.