I’m an elixir noob, but I’m actually surprised that the distributed story is not _better_. For example, a big thing about OTP is that it allows running many processes that communicate by message passing: but all these processes are _colocated_ to a given node, i would have expected that OTP (or genserver or whatever) handles clustering of machines, scheduling of workloads and routing of messages for me so that I can start processes and call them without caring about where they run.
What about “distributed systems” is elixir actually shining at? From my reading, it seems rather like a good option to _avoid_ distributing (thanks to the light processes and concurrency handling)
As others have commented it can do all this. What it doesn't do is automatically balance/place GenServers and processes across your cluster. This is up to you. The ways in which you distribute work across a cluster is full of trade-offs.
Need raft concensus? Paxos? Something else? Want to place work with consistent hashing? Want to hydrate/dehydrate with persistence vs memory only?
Those are all important decisions and the languages do not force you to one option.
The structure of the system is such that you can call any function on any node in the cluster.
In order to do it well, you need to understand your own abstractions. If we have a cluster are we better off passing every function around the cluster or running some locally? Is the network overhead worth it in every case or only in some?
It's up to your code to make those determinations about your application, but everything you describe is fairly simple to implement on the BEAM.
First, if you write process-concurrent code in Elixir/Erlang, it will run on all the cores in your machine. It's automatically locally scalable. The number of processes should range from some small multiple of the no. of cores (say, 10x cores), up to 100k - 1m on a big memory multicore machine.
Second, as the other comments say, Elixir/Erlang have very transparent built-in ways to address other nodes, and the programs will scale as expected: more cpus v. network data distribution cost. So good for small tasks that require lots of compute. The native clustering will run to a few dozen nodes over a trusted LAN.
But soon, you will want some layer managing the cluster, try Swarm:
Erlang/Elixir nodes can actually be rather trivially clustered, you can spawn processes on other nodes and transparently pass messages to processes on other nodes, including things like closures. It does not magically schedule workloads across the cluster, but there are libraries that can do so.
What about “distributed systems” is elixir actually shining at? From my reading, it seems rather like a good option to _avoid_ distributing (thanks to the light processes and concurrency handling)