Tangent, but I always had a different understanding of the “thundering herd” problem; that is, if a service is down for whatever reason, and it’s brought back online, it immediately grinds to a halt again because there are a bazillion requests waiting to be handled.
And the solution to this problem is to slowly, rate-limited, bring the service back online, rather than letting the whole thundering herd go through the door immediately.
That's really not the traditional meaning of thundering herd, which is about waking up all the processes when a connection comes in, then they all try to accept it and it's a lot of work for nothing. You get much better results if only a single process is woken up for each event.
Your problem is a real problem though. Where I worked, we would call that backlog, and we would manage it with 'floodgates' ... When the system is broken, close the gates, and you need to open them slowly.
In an ideal world, your system would self-regulate from dead to live, shedding load as necessary, but always making headway. But sometimes a little help is needed to avoid the feedback loop of timed out client requests that still get processed on the server keeping the server in overload.
Yea you are right. It could be a service being down and requests piling up, or a cache key expiring and many processes trying to regenerate the value at the same time, etc.
I think the article just used this phrase to describe something else. (Great article otherwise).
The short version is that when you have multiple processes waiting on listening sockets and a connection arrives, they all get woken up and scheduled to run, but only one will pick up the connection, and the rest have to go back to sleep. These futile wakeups can be a huge waste of CPU, so on systems without accept() scalability fixes, or with more tricky server configurations, the web server puts a lock around accept() to ensure only one process is woken up at a time.
The term (and the fix) dates back to the performance improvement work on Apache 1.3 in the mid-1990s.
Funny reading this comment after reading the article
> So many options meant plenty of levers to twist around, but the lack of clear documentation meant that we were frequently left guessing the true intention of a given flag.
And then reading your link, they complain >inside the docs< that the docs aren't complete. I have no idea what to believe anymore :D
The uWSGI docs also say, in the section called "uWSGI developers are fu*!ing cowards": "why --thunder-lock is not the default when multiprocess + multithread is requested? This is a good question with a simple answer: we are cowards who only care about money."
That's not the thundering herd. If someone rings the door (request), only one person (agent, process) needs to answer the door. But what might happen is that everyone in the house rushes to answer the door. The people "thundering" to the door (and making a mess as they do so) are the "herd". This can quickly become a problem if there are a lot of people in the house and the doorbell keeps ringing.
> but I always had a different understanding of the “thundering herd” problem; that is, if a service is down for whatever reason, and it’s brought back online, it immediately grinds to a halt again because there are a bazillion requests waiting to be handled.
That... doesn't have much to do with the thundering herd problem. It also doesn't make much sense as a concept on its own merits -- say you come in to work and your inbox is full enough for three inboxes. Does that fact, in itself, mean that you decide you're done for the day? No, it just means you have a much longer queue to work through than usual.
The thundering herd problem refers to what happens when (1) a bunch of agents come to you for something while you're busy; (2) you tell them all "I'm busy, go away and come back later"; and (3) the come-back-later time you give to each of them is identical, so they all come back simultaneously.
And that's exactly what's happening here, except that instead of giving each worker thread a come-back-later time when it asks for work, you're receiving work, sending out individual messages to every worker saying "hey, I'm not busy anymore, come back RIGHT NOW and get some more work", and then rejecting all but one of the thundering herd that shows up. The reason the Gunicorn docs and the uWSGI docs both refer to this as a "thundering herd" problem is that it's a near-perfect match for the problem prototype. The only difference is that, instead of giving out identical come-back-later times to worker threads as they ask you for work, you tell them to wait for a notification that includes a come-back-later time, and then when you get one piece of work you fire off that notification separately to every sleeping thread, including identical come-back-later times in each one.
> That... doesn't have much to do with the thundering herd problem. It also doesn't make much sense as a concept on its own merits -- say you come in to work and your inbox is full enough for three inboxes. Does that fact, in itself, mean that you decide you're done for the day? No, it just means you have a much longer queue to work through than usual.
If my SLA is 24 hour response time, and the inbox is FIFO, and I can't drop old messages, I'm most likely not hitting the SLA. If they all came in overnight, I'll hit the SLA for day 1, but I will be busy all of day 2 and 3 and never respond on time. If after day 1, I get a days worth of messages every day, I'll never catch up.
OK. But that's not a problem of a thundering herd. It's a problem that you have more incoming work than you are theoretically able to handle even if you stay in continuous operation. Your problem is solved by adding the capacity to do more work. The thundering herd is solved by purposefully desynchronizing incoming work requests.
Oh, I agree it's not thundering herd, but it is a real problem. Especially if you start getting retries after the first requests timed out. Some sort of backoff with jitter to avoid synchronized retries helps, but what really helps is dropping or not accepting requests when the processing will not be timely. That's simple to say, but not always simple to do.
Adding capacity is also simple to say, but not always simple to do. And there can be a large difference between the capacity needed to handle a cold start at peak vs the capacity needed for peak under regular operations.
This reminds me of inrush current when starting large motors... You get a huge current spike when you initially turn on the motor, so large that it can trip the breaker.
One solution is to use a soft starter which slow brings the motor up to speed.
Unfortunately HAProxy doesn't buffer requests*, which is necessary for a production deployment of gunicorn. And for anybody using AWS, ALB doesn't buffer requests either. Because of this I'm actually running both HAProxy and nginx in front of my gunicorn instances—nginx in front for request buffering and HAProxy behind that for queuing.
If anybody is interested, I've packaged both as Docker containers:
I don't think that's what I'm looking for, that's queuing at the front of the pipe but I need it queuing at the end of the pipe. Apache should be buffering and queuing lots of connections (with a timeout) and sending them single-file in gunicorn.
This is somewhat suspect. At my place of work, we operate a rather large Python API deployment (over an order of magnitude more QPS than the OP's post). However, our setup is... pretty simple. We only run nginx + gunicorn (gevent reactor), 1 master process + 1 worker per vCPU. In-front of that we have an envoy load-balancing tier that does p2c backend selection to each node. I actually think the nginx is pointless now that we're using envoy, so that'll probably go away soon.
Works amazingly well! We run our python API tier at 80% target CPU utilization.
So, in guincorn default mode (sync), the mode I'm assuming you're using. This means you really have 1 process handling 1 request at a time. The "thundering herd" problem really only applies to connection acceptance. Which is to say, that in the process of accepting a connection, it is possible to wake all idle processes that are waiting for a connection comes in (they will wake and hit EAGAIN and then go back to waiting.) Busy processes that are servicing requests (not waiting on the accept call) will not be woken, since they aren't waiting on a new request to come in. The "thundering herd" problem as I understand it, can indeed waste CPU cycles, but only on processes that aren't doing much anyways. I do however believe that `accept()` calls have been synchronized between processes on Linux for a while now to prevent spurious wakeups. You should verify you're actually doing spurious wakeups by using `strace` and seeing if you are seeing a bunch of `accept()` calls returning EAGAIN.
In gunicorn, `sync` mode does exhibit a rather pathological connection churn, because it does not support keep-alive. Generally, most load balancing layers already will do connection pooling to the upstream, meaning, your gunicorn processes won't really be accepting much connections after they've "warmed up". This doesn't apply in sync mode unfortunately :(. Connection churn can waste CPU.
Another thing to also note is that if you have 150 worker processes, but your load balancer only allows 50 connections per upstream, chances are 100 of your processes will be sitting there idle.
Something just doesn't feel quite right here.
EDIT: I do see mention of `gthread` worker - so you might be already able to support http-keepalives. If this is the case, then you should really have no big thundering herd problem after the LB establishes connections to all the workers.
Could the discrepancy be explained by the type of responses?
Sounds like an app like clubhouse might have lots of small, fast responses (like direct messaging), where very little of the response time is spent in application code. Does your API happen to do a lot of CPU-intensive stuff in application code?
HAProxy is a beautiful tool but it doesn't buffer requests that is why NGINX is recommended in front of gunicorn otherwise it's suspectible to slowloris attack. So either cloubhouse can be easily DDOS'd right now or they have some tricky setup that prevents slow post reqests reaching gunicorn. In the blog post they don't mention that problem while recommend others to try and replace NGINX with HAPRoxy.
> In fact, with some app-servers (e.g. most Ruby/Rack servers, most Python
servers, ...) the recommended setup is to put a fully buffering
webserver in front. Due to it's design, HAProxy can not fill this role
in all cases with arbitrarily large requests.
A year ago I was evaluating recent version of HAProxy as buffering web server and successfully run slowloris attack against it. Thus switching from NGINX is not a straightforward operation and your blog post should mention http-buffer-request option and slow client problem.
Performance is the only thing that is holding me back to consider Python for bigger web applications.
Of the 3 main languages for web dev these days - Python, PHP and Javascript - I like Python the most. But it is scary how slow the default runtime, CPython, is. Compared to PHP and Javascript, it crawls like a snake.
Pypy could be a solution as it seems to be about 6x faster on average.
Is anybody here using Pypy for Django?
Did Clubhouse document somewhere if they are using CPython or Pypy?
When using something like Golang, I have apps doing normal CRUD-ish queries at 10k QPS, on 32c/64g machines. For most web apps, 10k QPS is much more than they will ever see, and the fact that it is all done in a single process means you could do really cool things with in-memory datastructures.
Instead, every single web app is written as a distributed system, when almost none of them need to be, if they were written on a platform that didn't eat all of their resources.
I could rephrase you comment as why would anyone use Go when I could just use assembler or C and keep all into a single node.
People don't use python because they want performance. People use python because of productivity, frameworks, libraries, documentation, resources and ecosystems. Most projects don't even need 10k qps, but instead most projects do need an ORM, a migrations system, authentication, sessions, etc. Python has bottle tested tools and frameworks for this.
People have been taught to be irrationally afraid of in-process concurrency (including async). Not too long ago the standard approach for concurrency was "it's hard, don't do it".
I've been told off in code review for using Python's concurrent.futures.ThreadPoolExecutor to run some http requests (making the code finish N times faster, in a context where latency mattered) "because it's hard to reason about".
Backend controller performance is rarely a bottleneck, and if raw-compute still is there are a number of ways to speed it up, such as cython and/or work queues.
Typescript is a nicer language than Python in many ways and it doesn't suffer from Python's crippling performance issues or dubious static typing situation. Plus you can run it in a browser so there's only one language to learn.
Typescript would be nice if it weren't essentially just a bunch of macros for JavaScript. As it is now, as soon as you want to run it, you lose all the benefits of it (including many performance optimizations that could be made in a statically typed runtime) and of course, all the usual footguns of vanilla JS still apply. It's a great development tool though, I'll give you that.
> Which exacerbated another problem: uWSGI is so confusing. It’s amazing software, no doubt, but it ships with dozens and dozens of options you can tweak.
I am glad I am not the only one. I've had so many issues with setting up sockets, both with gevent and uWSGI, only to be left even more confused after reading the documentation.
If you’re delegating your load balancing to something else further up the stack and would prefer a simpler WSGI server than Gunicorn, Waitress is worth a look: https://github.com/pylons/waitress
Interesting to read that they are using Unix sockets to send traffic to their backend processes. I know that it's easily done when using HaProxy but I have never read about people using it.
I guess the fact that they are not using docker or another container runtime makes sockets rather simple to use.
> Python's model of running N separate processes for your app is not as unreasonable as people might have you believe! You can achieve reasonable results this way, with a little digging.
I have been through this journey, we eventually migrated to Golang and it saved a ton of money and firefighting time. Unfortunately, python community hasnt been able to remove GIL, it has its benefits (especially for single threaded programs), but I believe the cost (lack of concurrent abstractions. async/await doesn't cut it) far outweigh it.
Apart from what the article mentions, other low hanging fruits worth exploring are
[1] Moving under PyPy (this should give some perf for free)
[2] Bifurcate metadata and streaming if not already. All the django CRUD stuff could be one service, but the actual streaming should be separated to another service altogether.
I read the article and could not believe that was their takeaway. sometimes people are determined to vindicate their technology choices, no matter what.
Interesting to see this. It sounds like they're not on AWS, given that they mentioned that having 1000 instances for their production environment made them one of the bigger deployments on their hosting provider.
If not for the troubles they experienced with their hosting provider and managing deployments / cutting over traffic, it possibly could have been the cheaper option to just keep horizontally scaling vs putting in the time to investigate these issues. I'd also love to see some actual latency graphs, what's the P90 like at 25% CPU usage with a simple Gunicorn / gevent setup?
I was wondering that too, but there aren't that many common cloud provider that has 96 vCPU offering.
I am also wondering on 144 Workers, on 96 vCPU which is not 96 CPU Core but 96 CPU thread. So effectively 144 Workers on 48 CPU Core possibly running at sub 3Ghz Clock Speed. But it seems they got it to work out in the end. ( May be at the expense of latency )
Assuming you're running a system where normal request/response handling blocks on database queries it's often optimal to have more workers than available cpu threads and 1.5x is a common rule of thumb to try first.
I kinda get that honestly. It’s why I’ll spend $20 without even thinking for take out but not spend $2 for an app. It’s because the cost off the software is way way more than the money. It’s a commitment to actually use it and integrate it, deal with their sales team, talk to purchasing, handle licensing, and introducing friction to replacing it or using tools that don’t integrate well because “well we already pay for it.” Licensing also complicates deployments substantially when you’re doing lots of autoscaling.
And on top of that Nginx Plus is also expensive as hell.
Maybe for barebones compute they’re cost+ but I don’t think that’s really true for other services. For example traffic should cost effectively zero to them but they charge a huge premium. Some other managed services also appear to use value based pricing
Hm actually might be google based on what their traffic is going to (i only looked just now). Ok now it makes more sense why support wasn’t able to figure this out =)
If it is just a backend, why not port it over to one of the myriad of cloud autoscaling solutions that are out there?
The opportunity cost of spending time figuring out why only 29 workers are receiving requests over adding new features that generate more revenue, seems like a quick decision.
Personally, I just start off with that now in the first place, the development load isn't any greater and the solutions that are out there are quite good.
Author here. We do and did use autoscaling heavily but at a certain scale we just ran out of headroom on the smaller instance types we were using. Jumping to a much larger instance types meant that we will likely never run into those headroom issues again, plus solves other problems like faster spin up, better sidecar connection pooling and allows for a much higher hit rate on per instance caching.
You were autoscaling a single threaded process. You had 1000 connections coming in and scaling 1000 workers for those connections. Everything was filtered through gunicorn and nginx, which just adds additional latencies and complexity, for no real benefit.
What I'm talking about is just pointing at something like AppEngine, Cloud Functions, etc... (or whatever solution AWS has that is similar) and being done with it. I'm talking about not running your own infrastructure, at all. Let AWS and Google be your devops so that you can focus on building features.
According to the article they have a monolithic Django application so this will have at least a couple of seconds start-up time. That is not a good match for Cloud Functions.
Django also has in-memory caches, for example for templates which can be extremely slow (seconds) and CPU intensive to render. So you really don't want to have AWS or Google restart your application on AppEngine whenever they feel like it.
There's a few reasons why this scenario wouldn't be a good fit for cloud functions, but that "couple of seconds start-up time" can be almost entirely removed from the equation by keeping the Django instance alive (all cloud function type offerings will have a concept of cold and warm starts, and some way to control persistence across calls on the same "instance").
I've run Django on AWS Lambda in a a scenario that scaled between 25-250 calls per second depending on time of day (for a runtime of 5-30 sec). Moving Django's bootstrapping so it would stay warm across calls was very easy.
> unless you have unfixable memory leaks there is no reason to do this.
It's also useful to set this threshold to prevent long-lived connections to services/datastores not used by every request from accumulating and consuming resources on those services.
a) you get to fire the devops person, which saves $150k+ a year.
b) you add appropriate caching layers in front of everything.
c) you spend time adding features, which generate revenue.
I've done all of this before at scale. This whole case study was written about work I did [1]. Two devs, 3 months to release, first year was $80m gross revenue on $500/month cloud bills. Infinite scalability, zero devops.
> you get to fire the devops person, which saves $150k+ a year.
You are deluded or extremely short-sighted if you believe you can actually fire the devops guy. From my experience, the more you stray away from the conventional "dedicated server" paradigm the more you need a devops guy and you are in a very precarious position if you do fire him and something goes wrong.
You don't hire the devops person until you've scaled to the point that you need one.
Additionally, your thought of having my company held hostage by a single devops person is terrifying. Now you need two of them, which is even more expensive.
It is a great way to bootstrap a company by saving on a salary (or two) that can honestly be engineered out for a lot of SASS businesses. It worked super well for us... and calling someone who did $80m in the first year deluded seems well, rude.
But, if you start off designing systems that scale on their own, you are much better prepared for when you do get some fast growth than dealing with hiring a good devops person (which is extremely hard, as they say.. all the good ones are taken).
At the end of the day, the actual elephant in the room is that django was the wrong choice. You end up having to go through a lot of contortions to make things work, as evidenced by the blog post. The architecture doesn't make things easy to spin up quickly... which creates a lot of bottlenecks. There are better cloud-based solutions.
If you don’t have a devops person, then you end up with developers pitching in to fill that void. That’s OK and may be desirable but it is still a cost.
They are on a back-end that does auto-scaling. They stated that they had problems when scaling up past 1000 nodes.
Now, maybe they could have fixed that issue instead, but going from 29 to 58 workers is easy, it's not the same going to 29,000 to 58,000. And 1000 hosts vs 500 is a non-trivial cost.
You'd probably be worrying more about instance sizes if you ran a single executor per container; the memory overhead of your app would become a problem very quickly unless it's startup footprint was quite small.
This doesn’t work so easily with architectures with process pools for workers. So now your app server needs to speak docker (or whatever control plane) to spawn new workers and deal with more complicated IPC. Also the startup time is brutal.
One process per container and multiprocessing is a huge lift most of the time. I’ve done it but it can be a mess because you don’t really have as much a handle on containers than subprocesses because you can only poke them at a distance through the control plane.
Do you mean multiprocessing inside the containers? Or are you managing multiprocessing child procs by forking into a container somehow? If the latter, I'd be really interested to learn how to do that; I didn't think it was possible, and it would be super useful for some of what I work on.
Knowing nothing else, it's hard to know if this is good or not. It's 16 requests per second. Are those requests something like "Render a support article" or are they "Give the user a ranked feed of what they should see on their home screen"? Is most of the logic run by the web server or some combination of app servers / backend services behind it? What kind of hardware does the web server have?
All of those would affect the answer, and would preclude being able to guarantee "up this by couple orders of magnitude"
Well, to give you an idea, I am working on an service that implements rather complex business process involving fetching data from multiple sources, parsing binary blobs with market data in proprietary format, saving results to a database and so on. And it does around 10k requests per second on a single relatively normal node (8 cores, 64GB ram, etc.)
And no, it does not require any special tricks. It is regular Java / WebFlux / REST / MongoDB backend service.
CPUs can do really a lot and if your node processes 16 requests per second on a multi-core machine then you are using billions of clock cycles and gigabytes of possible transfer to memory for a single request. Something is not quite right...
As a somewhat imprecise example, if a single "request" requires sorting 2.4 billion integers, then a 2.4 GHz CPU with 16 cores will be able to process at most 16 RPS no matter how much you switch from JavaScript to Java or if you write assembly.
At the end of the day efficiency is ultimately a business problem and not a technical problem and is rarely the thing that tips a project (Clubhouse in the article) from being profitable to being unprofitable. It's usually an investing question - I have X engineer-months to spend. I can cut costs by Y by optimizing stuff or get Z more profit by building a feature. I will choose to optimize stuff if and only if Y>Z as it returns more.
Clubhouse's major costs are probably bandwidth and engineer time rather than servers. That is to say, even if efficiency was infinity for compute (i.e. server costs magically went to zero) it would probably not change Clubhouse's business proposition that much.
More to the point, I think you are uncharitable at best when you say elsewhere that other frameworks and languages won't require more development work. These frameworks (and the choice of language being implicit in that) are specifically designed to reduce development work. Let's examine for example garbage collection. Garbage collection is undeniably more wasteful than other solutions to memory management, absolutely. But would you really argue that garbage collection does nothing to reduce development time? I find that extremely hard to believe, empirically and subjectively having written programs in many environments including bare metal, reference counted or otherwise semi-managed and garbage collected languages. And so it goes with all of the choices these frameworks like Django and Rails take. And it's getting better with time as things like JRuby are developed, inefficiencies in Rails or Django are removed, etc.
This is a comically yet incredibly common engineering bad take. When you run a company there is only one question to answer, one north star - does it make money ?
To be honest the article does realize this, first blaming it on the poor hindsight from original developer (co-founder) and in the conclusion about maybe rewriting the whole thing.
It seemed to be all about how to extract the most performance from the lemon they had to deal with.
I don't know Python or how complex their domain is but the number of workers suggests to me it is not that complex and their application spends most of its time switching contexts and in inefficient frameworks.
Per my experience most applications that mostly serve documents from databases should be able to take on at least 10k requests per second on a single node. this is 600k requests per minute on one node, compared to their 1M per 1000 nodes.
This is what I am typically getting from a simple setup with Java, WebFlux and MongoDB with a little bit of experience on what stupid things not to do but without spending much time fine tuning anything.
I think bragging about performance improvements when your design and architecture is already completely broken is at the very least embarrassing.
> poor hindsight from original developer (co-founder)
Well, you have a choice of technologies to write your application in, why chose one that sucks so much when there are so many others that suck less?
It is not poor choice, it is lack of competency.
You are co-founder and want your product to succeed? Don't do stupid shit like choosing stack that already makes reaching your goal very hard.
The job of the cofounder is to create a thing that people want, which has nothing to do with performance. The first goal is capturing lightning in a bottle with social products. Performance doesn’t matter until the lightning is there, and 99%+ of the time you never have to worry about performance, because you don’t get the lightning. So, probably the correct choice is leveraging the tech stack that gives you the best shot at capturing the lightning. Django seemed to help!
Plus, if he could've predicted the pandemic that far in advance there would probably have been plenty of not clubhouse ways to monetise that prescience ;)
The job of the cofounder is also to anticipate possible risks.
And building your company on an astronomically inefficient technology sounds like a huge risk to me.
Those 1000s of servers are probably a very significant cost with such small technical staff. Just by choosing the right technology for the problem, most of that cost could have been avoided.
Django has nothing special in it that would allow building applications faster than in a lot other frameworks that are also much more efficient.
So it is just a matter of simple choice.
Nobody expects people to write webapps in C++ or Rust. Just don't choose technology that is famous for being inefficient.
Python is not astronomically inefficient. Instagram serves like a billion users with it. Job of a cofounder is to build what people want. You can always scale in Silicon Valley by hiring people like you. You can’t build another viral app like clubhouse by hiring from the same crowd.
This may hurt you but the truth is scaling and software engineering is highly commoditised. That’s the whole point of being in the valley. You can hire people for such things and forget about it.
Clubhouse is not a tech company. They don’t have to care about being the best at infra
When you spin 1000s of nodes you need some tech competency.
Or in other words, if it blew one day and there would be a link to writeup on HN, people would be asking "They had 1000s of servers and nobody competent to maintain it?"
You sound just like the average sports fan commenting after a match about what x player should have done, shouldn't have done, blame it on decisions, style of the trainer, owner etc..
But you're just that.. a fan yapping about how they could do better.
It is the decision to choose it to run load that will require 1000s of servers when it could be handled with 5-10 servers in another technology without more development effort.
I doubt they expected that level of request load that early on - I imagine the technology choice was made significantly before the whole pandemic thing started.
Yes all of those would be way better options than Python and probably PHP. Well maybe not C++. You'd have to be pretty crazy to have web developers writing security sensitive code in C++.
The "blame our co-founder for the choice" bit is exactly what that graph about the cost of defects vs how early they are fixed is talking about.
If they had just picked Go or Java right at the start they wouldn't have had to expend all this engineering effort to get to a still-not-very-good solution.
Java wins as expected, but a typical setup with Spring versus the typical top PHP frameworks isn't blowing the doors off. Typical Python + Django is far behind, as someone pointed out.
However what we can see in the diagrams is that ORM layers, regardless of language, are more expensive than what most people realize, even for a compiled language like Java.
Why PHP wins is because it is fast enough, compared to other dynamic languages, but is a better fit for web development than Java or other compiled languages.
That is for spring-webflux, is that a typical spring set up for web today?
I haven’t coded spring for a few years now, but
I was thinking about the traditional spring setup that most use and that is comparable.
You can of course use Python successfully, my argument is that it is easier with PHP, not that it is not possible with Python.
It is a similar argument compared with Java, it is easier with PHP than Java in a web context. Java has other benefit thats fits better for web services IMHO, general higher performance is one.
Serverless is too overloaded a term to have any meaning. I'm not really seeing how Python or PHP "scales infinitely" in any way that C#, Java, C++ couldn't.
PHP is usually easier to scale because it just a matter of how many webservers. e.g. apache or nginx, you choose to deploy.
This also possible with other platforms, but can be a bit trickier to get right.
For large PHP setups it is usually the number of database connections that is the limiting factor, however that is why historically the replicated MySQL databases was such a good fit for PHP, thus only creating a limit for writes on the master.
> For large PHP setups it is usually the number of database connections that is the limiting factor,
For pretty much every modern programming language, IO is the bottleneck over everything else.
To save you some time, there are practically no metrics in which I think PHP beats another programming language other than maturity, and even then, not really.
Yet YouTube, Instagram, Pinterest, Reddit, Robinhood, DoorDash, and Lyft backend were originally primarily written in Python. What’s funny is that nobody can really deny Python is slow yet somehow the biggest websites in the world were written in it. More proof that Worse Is Better?
Famous last words, but I get the sense that the need to handle this sort of load on Clubhouse is plateauing and will decline from here. The app seems to have shed all the people that drew other people initially and lost its small, intimate feel and has turned into either crowded rooms where no one can say anything, or hyper specific rooms where no one has anything to say.
Good article though! I’ve dealt with these exact issues and they can be very frustrating.
I wouldn't be very proud of writing an article like that.
Usually engineering blogs exists to show that there are fun stuff to do in a company. But here it just seems they have no idea, what they are doing. Which is fine, I'm classifying myself in the same category.
Reading the article I don't feel like they have solved their issue, they just created more future problems
And the solution to this problem is to slowly, rate-limited, bring the service back online, rather than letting the whole thundering herd go through the door immediately.