Note that you not only have to worry about FFI bindings with this sort of architecture (as mentioned in the article), but also blocking syscalls as it'll freeze all the other processes running on the thread. You end up having to re-invent a lot of stuff otherwise provided by the OS, which is why Rust replaced its green thread architecture with native threads by default [1].
This doesn't have to be true (though working around it is admittedly difficult) and I'm really disappointed to hear about the replacement. You can't spawn 10k native threads without grinding your application to a halt, which is the entire point of "green" threads.
> but also blocking syscalls as it'll freeze all the other processes running on the thread.
I think he was talking about Erlang, do you know what system calls freezes all the other processes in Erlang? I know it was engineered to have an I/O thread pool and scheduler specifically to handle those calls. So it seems that is not necessarily true (although it maybe true for Rust).
> Note that you not only have to worry about FFI bindings with this sort of architecture (as mentioned in the article)
Once you start writing C bindings and loading them into directly into the VM memory space you lose the isolation and many things that come with it. There are other ways to talk to C -- use a port (spawn as sub-process and use stdin/stdout + serialization, there is helper library to help with that code). Can also emulate whole Erlang node (VM) from C and talk to it using distributed Erlang protocol via a socket.
Other things that wreck your systems are running out of memory, out of disk space, filling your mailboxes or message queues due to large spike or consistent rate mismatches between senders and receivers. Also deadlocks.
The number #1 thing that wrecks systems is ... software bugs mixed with complexity. Throw concurrency, shared memory and pointers on top and it becomes a disaster. Small demos still look and work very very fast. Large concurrent systems become a mess. Yeah everyone can hold their hand up and swear to never create global structures or shared pointers and always talk via queues/mailboxes but then so did application when they ran in Windows 3.1 or DOS environment. It worked ok for that time. But eventually your word processor would crash because your game overwrote its memory and so on.
Running concurrent units of execution (threads), mixed with pointers and a shared memory is like going back and time and running Windows 3.1.
In many ways this reminds me of the L4 project [1] - very lightweight "shared nothing" processes, semi-cooperatively scheduled [2] and with attention paid to minimising the context switch overhead. And this was before "shared nothing" was a thing. It's a real shame that so few of Jochen Liedtke's ideas have percolated into mainstream OSses.
[2] IIRC, a source process sending a message would 'donate' the rest of its time-slice to the destination process, meaning short "RPC" calls could be done without invoking the scheduler while keeping many of the benefits of pre-emptive scheduling.
A point often missed is that for green threads to work really well you have to pull most of what the operating system does into user space to eliminate blocking syscalls. That is a lot of work but if you do it competently it will greatly outperform (by integer factors) the same design going through the operating system. This is how high-performance database engines are designed.
As with all such low-level hackery you really need to understand what you are doing to get a good result, but if you do understand what you doing the results can be excellent.
This is one of the reasons for MIT's exokernel design- most of what the operating system does gets pulled into userspace libraries by default, the same way it is with green threads, so you get the same kinds of benefits with file systems, networking, IPC, etc.
I'm kind of disappointed most kernels that have added scheduler activations have dropped support because they didn't want to deal with both interfaces and weren't willing to drop the traditional one.
1: https://cmr.github.io/blog/2014/03/24/this-week-in-rust/