Agreed, if you haven't tested your backups recently (daily, automatic best), you don't have backups. Several of my clients (CTO Coaching) had problems in the past because they restored backups and where finding they were not complete (for various reasons).
daily test restore is infeasible for anything but toy projects.
You should periodically test your restore procedures, but its incredibly costly and time consuming for sizeable platforms. Its just not that easy to restore a 10+TB backup for example, and thats a _tiny_ backup size for a b2c product.
they can easily go into the hundreds of TB, depending on your platform.
and i might add: i vividly remember gitlabs article how they have had automated backups and test restores for years, but when they actually needed them... it turned out some data wasn't part of it after all. just because youre testing your restore procedure doesnt mean you've actually accomplished anything.
Have two backups, the most recent data and everything else. Archive data to different databases, e.g. only have the most recent 6 months in a production OLTP database.
"daily test restore is infeasible for anything but toy projects. "
It probably depends on what you call "toy" project. If you work for Google, yes I think everything is a toy project, and you're right. I only worked for ~$200M ARR/1M DAU businesses and restoring was no problem. From your point working for a FAANG business it's a toy project I can see that. But there are many more "toy projects" of this kind than FAANG companies.
"10+TB backup for example, and thats a _tiny_ backup size for a b2c product."
Why is that? Hours or even days of downtime are still better than just losing all data. It's a simple cost-benefit analysis, and it's ok to pick different trade-offs depending on your use case
Russia and Iran would like to have a few words with you.
Banks regularly close for multiple days for bank holidays. Unscheduled downtime is a somewhat different story, though.
Luckily, the traditional SWIFT/banking infrastructure is so negligible these days, my phone can host a classic banking infrastructure for an entire small country.
A bank exists not as an isolated entitity, but as a node in a local, regional and global network of transactions.
Your phone as a “classic banking infrastructure” (nonsense phrase) can’t do credit card acquiring or realtime transactions because it’s not connected to the payment rails, transaction switches and so on (like SWIFT but all of the other less global ones run by central banks and private entities).
In developed societies, instant settlement for bank-bank transfers is the norm, and cash flow is dependent on that.
Russia and Iran pay about 2-5% for above-board (non-sanctioned) cross border transactions due to their extra costs of not being in SWIFT and USD sanctions, and between 20-50% where physical middle-people are needed to move pallets of USD.
