Sequencing in distributed systems is difficult to handle correctly. When faced with such a situation, it’s often a matter of philosophy rather than technical brilliance - are you willing to accept that failure is inevitable even with a neat “waiter” implementation? What happens if the waiter waits forever? Is shaving seconds tangibly benefitting customers (for Sentry it might) and worth the operational overhead?
Framing these problems slightly differently - is 10 second latency close enough to “real time”? - unlocks solutions like an online batch processor that reads out of your DB (if it doesn’t support NOTIFY) on an indexed incremental identifier.
I'm not sure about the first part of the solution. Why isn't it enough for the "post process" consumer to wait for events on a "persisted events" topic? And have snuba copy there a message only after it was persisted (and available on all replicas)?
/edit: actually, why does the "post process" consumer even cares about events not persisted yet?
Framing these problems slightly differently - is 10 second latency close enough to “real time”? - unlocks solutions like an online batch processor that reads out of your DB (if it doesn’t support NOTIFY) on an indexed incremental identifier.