If you have a lot of connections doing similar things, just batch requests to get the data in bulk.
Scaling your database up should only be attempted once you can no longer improve efficiency of your application. It is always better to first put effort into improving efficiency than scaling it up.
For example, one trick that allowed me to improve throughput of one application using MongoDB as a backend by factor of 50 was capturing queries from multiple requests happening at the same time and sending them as one request (statement) into the database, then when you get the result you fan them out to the respective business logic that needs them. The application was written with Reactor which makes this much easier than a normal thread based request processing.
For example, if you have 500 people logging at the same time and fetching their user details, batch those requests for example every 100ms up to 100 users and fetch 100 records with a single query.
You will notice that executing a simple fetch by id query even for hundreds of ids will only cost couple times more than fetching a single record.
The application in question was able to fetch 2-3 GB of small documents per second during normal traffic (not an idealised performance test) with just couple dozen connections.
Can you please edit swipes like "you are doing it wrong" out of your HN comments? It acidifies the thread and evokes worse from others. The site guidelines are pretty clear about this: https://news.ycombinator.com/newsguidelines.html.
Edited out but the issue is it really is technically wrong. From a person that is actually building real world large scale applications this is super naive. There exists no networking hardware or disk IO that can adequately support this many active connections. What you really want to do is to do the same work with much less connections which will save you a ton of overhead and also memory that can be used to actually cache useful data.
My goal is to share this little bit of insight so that hopefully fewer people make this fatal mistake. Just because you can technically do something does not mean this is a good idea. If you need to run a million connections to the database instance you really need to rethink this entire problem.
I’m not sure I’d say anything is a “fatal mistake” if it works, and this appears to? Feels like you’re coming at this from “it’s suboptimal”, calling it a fatal mistake is a bit much?
I would look at it this way:
Firstly, it’s just a demonstration of how far a technology can scale, not an endorsement of technical approach for the customers.
Secondly, on serverless, PlanetScale is just solving things their customers want? They don’t have a business if they don’t do that.
Finally, even if you are “right” or at least making a valid point, isn’t the “rethink this entire problem” take principled more than practical? There’s always 47 things to do and time only exists to do a few.
How could you possibly say this is doing it wrong? The only way you could batch requests in the way you describe is if you have 1 (or very small number) compute nodes. You would need all those requests to hit same node so you could try and batch. With serverless compute infrastructure (which is what this blog is demonstrating by using lambda) you can have 1 isolated process per request and therefore need a database that can actually handle this kind of load.
Here is your problem. You are trying to build a huge application using inadequate technical building blocks.
Lambdas are super inefficient in many different ways. It is a good tool but as with every tool you do need to know how to use it. If you try to build heavy compute app in Python and then complain at your electricity bill -- that really is on you.
If your database is overloaded with hundreds of thousands of connections from your lambdas, it means it is end of the road for your lambdas. Do not put effort into scaling your database up, put effort into reducing the number of your connections and efficiency of your application.
I think you can start to hit connection limit walls with RDS at several hundred connections, depending on your instance size. Running an even moderately busy app you could hit those pretty quickly. I would hate to have to change my entire infrastructure at such an early stage because the DB was hitting connection limits!
Would you ever need a million open connections? Probably not! But you'll likely want more than 500 at some point. And if your entire stack is serverless already, it'd be nice if the DB could handle that relatively low number of connections too.
I look at the database connections the following way: how many connections can a database really serve effectively? For a connection to be actively served the database really needs to have a cpu core working on it or waiting for IO from the storage. And I am completely omitting the fact that databases really need a sizeable amount of memory to be able to do things efficiently.
Even if you have a server with hundreds of cores your database probably can't be actively working on more than a small multiple of the number of the cores.
I am not saying you can't. I totally believe you do.
Modern hardware is totally able to execute hundreds of thousands of transactions per second on a single core. If your query is simple and you can organise getting the data from storage at the necessary speeds you should totally be able to do this many requests, possibly even tens of millions.
But handling one million queries per second is completely different from having database server making progress on one million queries in parallel. What happens is, the database server is only making progress on a small number of them (typically in tens up to hundreds on a very beefy hardware) and everything else is just queued up.
There are much, much better ways to queue up millions of things than opening a million connections to get each one done individually.
Lambda was a means to an end for us here, and we're not specifically endorsing its use in _this_ way. Our goal was explicitly to test our ability to handle many parallel connections, and to observe what that looked like from different angles.
We're a DBaaS company, and we do need to be prepared for anything users may throw at us. Our Global Routing infrastructure has seen some major upgrades/changes recently to help support new features like PlanetScale Connect and our serverless drivers.
From our point of view, this was a sizing exercise with the interesting side benefit that many people do happen to use Serverless Functions similarly.
How much is a moderately busy app? I have a sketch of a twitter app in Scala with zio-http as the framework, doing the batching strategy twawaaay describes, and it can handle 46k POSTs per second on my i5-6600 with a SATA3 SSD. That's using 16 connections to postgres, which is probably more connections than is reasonable for my 4 core CPU.
At 46k RPS, it only takes 5.5 ms to assemble a batch of 256, so latency is basically unaffected by doing this. Just set a limit of 5-10 ms to assemble the batch (or lower if you have a more powerful computer that can handle more throughput).
But then you can't put "led implementation of a highly-scalable $80m/yr cloud-native solution" on your resume. And bigger numbers mean bigger salary.
Now, if you wait for someone else to do this then you swoop in and cut that spending by 90%... now you're talking. Then the next person can reverse what you did and we all keep making money digging holes then filling them back up.
Yeah, it is better to manage 50 people and millions in hardware than do the same work with just 2 engineers and one server.
You then get those completely unnecessary feats of technology when really, a single server is usually good to run hundreds of thousands of transactions per second.
It infuriates me that incompetence is promoted and the only way to get rewarded is to be good at internal politics.
Just wanted to come back and defend you (a little :-)
I read "if you have a million database connections then you are doing it wrong" less as a ad hominem attack and more as "if one is doing X one should try a different approach"
I did like the "architectural strategy" of I can call it that of batching the calls. It's "tricks" like that, expressed at this level that are somehow missing from the common software dev parlance. They are not in l33tcode tests, they don't fit into neat boxes but they are vital "common knowledge"
I wish I had a better term for these sort of optimisations.
Anyway. Thanks for the comment. Don't take the blowback personally - frankly I was surprised even if it was a small storm in a teacup.
Scaling your database up should only be attempted once you can no longer improve efficiency of your application. It is always better to first put effort into improving efficiency than scaling it up.
For example, one trick that allowed me to improve throughput of one application using MongoDB as a backend by factor of 50 was capturing queries from multiple requests happening at the same time and sending them as one request (statement) into the database, then when you get the result you fan them out to the respective business logic that needs them. The application was written with Reactor which makes this much easier than a normal thread based request processing.
For example, if you have 500 people logging at the same time and fetching their user details, batch those requests for example every 100ms up to 100 users and fetch 100 records with a single query.
You will notice that executing a simple fetch by id query even for hundreds of ids will only cost couple times more than fetching a single record.
The application in question was able to fetch 2-3 GB of small documents per second during normal traffic (not an idealised performance test) with just couple dozen connections.