Author here! Overcommit is definitely a thing to watch out for. I believe TigerBeetle calls this out in their documentation. I think you'd have to explicitly disable it on Linux.
For the second question, yes, we have to keep track of what's in use. The keys and values are allocated via a memory pool that uses a free-list to keep track of what's available. When a request to add a key/value pair comes in, we first check if we have space (i.e. available buffers) in both the key pool and value pool. Once those are marked as "reserved", the free-list kind of forgets about them until the buffer is released back into the pool. Hopefully that helps!
I have a few cases in this (proof of concept) codebase that require knowledge about allocation strategy, even in Zig, but that's on me and the design at this point. Something I wanted to touch on more in the post was the attempt to make the components of the system work with any kind of allocation strategy. I see a common thing in Zig projects today where something like `gpa: std.mem.Allocator` or even `arena: std.mem.Allocator` is used to signal intent, even though the allocator interface is generic.
Author here! That's totally fair. I did learn this is a common technique in the embedded world and I had a whole section in the original draft about how it's not a super well-known technique in the typical "backend web server" world, but I wanted to keep the length of the post down so I cut that out. I think there's a lot we can learn from embedded code, especially around performance.
Back in 2005 Virgil I, which target MCUs like AVR, had static initialization and would generate a C program with all of the heap statically allocated, which was then compiled into the binary. C programmers for AVR are used to just declaring globals, but Virgil allowed arbitrary code to run which just initialized a heap.
Virgil II and III inherited this. It's a standard part of a Virgil program that its components and top-level initializers run at compile time, and the resulting heap is then optimized and serialized into the binary. It doesn't require passing allocators around, it's just part of how the language works.
Limits in systems like these are generally good. They mention the reasoning around it explicitly. It just seems like the handling of that limit is what failed and was missed in review.
We didn't rule out using Tigerbeetle, but the drop in non-batch performance was disappointing and a reason we haven't prioritised switching our transaction ledger from PostgreSQL to Tigerbeetle.
There was also poor Ruby support for Tigerbeetle at the time, but that has improved recently and there is now a (3rd party) Ruby client: https://github.com/antstorm/tigerbeetle-ruby/
I think the drop in non-batch performance was more a function of the PoC than of TB. Would love to see what our team could do for you here! Feel free to reach out to [email protected]
It seems to me that, in practice, you'd want the "LiveBatcher" to have some durability as well. Is there a scenario where a customer could lose their place because of a horribly timed server shutdown, where those transfers hadn't even been sent to TigerBeetle as pending yet? Or am I misunderstanding the architecture here?
Edit: Yes, I think I misunderstood something here. The user wouldn't even see their request as having returned a valid "pending" ticket sale since the batcher would be active as the request is active. The request won't return until its own transfer had been sent off to TigerBeetle as pending.
I really like the idea of more decentralized git collaboration. What do people think are the biggest blockers to adoption in this space? Having to run a server or manage some kind of private keys? Is it purely network effect?
Spam, illegal content, and moderation in general. How do you protect against new account spam when any domain could be a PDS and any PDS could host an arbitrary number of users? What do you do about people stuffing ebooks and TV shows in git repositories? If a project is getting piled on with all its issued spammed because of political views of the repo's maintainer, is this considered a problem, and if so how is it fixed?
The advantage of an AppView is that, like BlueSky, you can actually have a central moderation team and consistent moderation policy. Even if people post whatever they want on their own PDS it is possible to curate what people normally see. However, even though I avoid following the drama I can see that the BlueSky moderation team is constantly under fire for some decision or other. Choose your poison.
Nowadays I don't have the appetite for fully decentralised public networks and all the responsibilities and problems they bring. It's nice that AT's content is completely open compared with something like Twitter, but it's so helpful that the day-to-day administration is centralised when you want an authority to appeal to without ending up with the quagmire of "defederation".
A question to ponder: is anyone here going to volunteer to run a "permissive" radicle seed node? (i.e. providing storage and access to arbitrary git repos uploaded anonymously)
But doesn't the decentralized firehose make it easy to build curation? You decide what/whom you want to subscribe to---rest of social media be damned. Why do you care what unmoderated crap is flooding the world outside your cosy corner?
And if you choose to receive a broader sampling, you can subscribe to someone who will curate it for you---either manually, or through algorithms. It seems like an elegant way to have a web-of-trust layer for curation, composed with an algorithmic curation layer---and be able to tune the latter separately to suit user needs, without being beholden to the interests of the platform operator. You can easily switch your subscriptions if you don't like the way someone is curating it, without wholesale losing access to the network!
> A question to ponder: is anyone here going to volunteer to run a "permissive" radicle seed node?
Doesn't opening up curation+subscription solve this problem too? Anyone can curate in opinionated ways, and offer to "host" whatever they are okay with accepting responsibility for (at whatever level of endorsement, so long as it is clearly communicated) and users have the choice to subscribe.
The problem today is that curation is tangled with access to the network, so you're forced to accept the curation provided to you by the owner of the walled garden (and incentives are misaligned)
AtProto does have platform and user managed labelers for the moderation piece, so it's at least built into the protocol. The jury is still out on how well that concept will scale.
Tangled very nicely gets rid of the having to run a server problem, yet still gives you a sovereign platform for doing git from. Truly divine.
The barrier is largely that Tangled is so new. People don't know about it. People don't know what Bsky & the Personal Data Server offer or they haven't been enticed out of the zero energy local minima.
There's some need for more features. For more tangled dev. Ideally for alternate clients, just because. But it's already enormously solid, the early adopters are living the better life, the future is already here and we are just waiting for devs to redistribute themselves appropriately.
Hi! Knots just serve up git repositories over an XRPC API. The actual state on disk is really just a sqlite + your git bare repositories—the two can be tarballed and moved elsewhere easily!
We will work on more first party backup/migrate solutions though.
Git repos don't get replicated anywhere. They live on disk, either on our knot server or yours. Knots are essentially our extension to the AT architecture, allowing user ownership of what's essentially "off-protocol data" (git).
Sure, you can mooch off their centralized server for a while. In an actually decentralized network there have to be multiple servers which cost money to run and that money is never recouped.
reply