As an aside this is something people absolutely do not get. MongoDB haters were always quick to point out how the database can lose data; even after the journal was introduced and that became much harder. But MongoDB failed to explain this same thing well: if you are storing payroll information you need a very robust data storage solution. There's a cost to robustness, however, which you probably don't need to pay when storing 140 character messages. Write concern is a very fundamental business decision.
In other words, when I picked MongoDB to store NowPublic SCAN data, before the word NoSQL was in vogue, even, then it was scanning various streams (including the Twitter gardenhose so many years ago) for bits of news and the ability to store lots and lots of small bits of text quick as hell was much more important than keeping all of them. Noone really cared about every single one, we were in the business of surfacing breaking news and we were a very small startup so not needing lots of expensive servers were a huge plus.
Obviously I picked a very different data storage model when I needed to store payment data at a TV station's video store...
We used Facebook as an example all the time. For them, moving fast and breaking things is a good value. If someone's status update got lost, that isn't ideal, but it isn't the end of the world.
When people don't get paid, that's a Big F-ing Deal.
It's an easy trap to fall into as an engineer. There are many cool problems to solve, but when you join a team you're agreeing to solve their problems. Focus on the customer.
In other words, when I picked MongoDB to store NowPublic SCAN data, before the word NoSQL was in vogue, even, then it was scanning various streams (including the Twitter gardenhose so many years ago) for bits of news and the ability to store lots and lots of small bits of text quick as hell was much more important than keeping all of them. Noone really cared about every single one, we were in the business of surfacing breaking news and we were a very small startup so not needing lots of expensive servers were a huge plus.
Obviously I picked a very different data storage model when I needed to store payment data at a TV station's video store...