Sure, but just as a thought experiment imagine you have a really bad disease. I give you two options: 100 high-school students can diagnose and prescribe a treatment or you can choose 1 professional with 10 years of experience in related diseases.
You might initially prefer the professional with 10 years of experience. Would your opinion change if I told you that I selected the high-schoolers to be diverse so that one is pessimist, one is an optimist, one got good grades in engineering, one loves philosophy?
Of course, my intuition might be wrong. For example, perhaps it legitimately would be better to have 100 high-school students where one is a high-school level ability with ear-nose-throat, one is a high-school level ability in oncology, one is a high-school level ability in cardiology, etc. Except some control AI would have to synthesize their answers into a coherent response ... and that controller would be high school level.
I'm not really sold either way if I am honest. I don't think we have the answers to these questions. It just shows that my intuition about intelligence is open to challenges.
In real life, though, that specialist with 10 years experience has a 2 year waiting list for new patients, and you wouldn't even get in to see them anyway, because they don't take your health insurance.
Meanwhile, the "100 high school students" on a mobile phone are the only medical consultation someone in Sub-Saharan Africa or Southern Asia is going to have access to at all.
"Why Not Both" would surely apply here. Maybe we can't give everyone access to a kind, caring, patient human doctor; but we sure can make coming in second place a lot less painful.
I suspect the divide may end up similar to that between a start up and a large company. Yes, some problems are better solved by a single person. But others benefit from a division of thought even if there is some bureaucratic overhead.
That is my speculation. But the question is one of intuition. If the speculation is correct, that some problems are better solved by a single more intelligent agent, then how can I determine the appropriate approach?
What I mean to say is, if I am considering building a product based on LLMs then I may have to make a basic decision: can I use multiple cheap LLMs in a multi-agent setup or must I use a single expensive powerful LLM. Right now I don't have any intuition on what kinds of problems are solved most efficiently by either approach. Just looking at a problem description I can't intuit which approach is appropriate.
This is a very interesting and important question. I think that until we answer your question rigorously, the best bet right now is to start with the expensive powerful LLM, and then after your system is prototyped and working, piecewise move prompts in your prompt network over to other systems as it proves feasible.
If you try to start with the multiple cheap LLMs approach, it is almost certainly going to be more difficult to get the prompting right, and if you don't already know what you are trying to accomplish is feasible, you're adding a lot of work that might add up to nothing (even if it would actually work if you used the more powerful model).
It's not quite the same thing, but the "mixture of experts" model is a popular deep learning approach that displays very good performance: see e.g., https://arxiv.org/abs/2208.02813
Sure, but just as a thought experiment imagine you have a really bad disease. I give you two options: 100 high-school students can diagnose and prescribe a treatment or you can choose 1 professional with 10 years of experience in related diseases.
You might initially prefer the professional with 10 years of experience. Would your opinion change if I told you that I selected the high-schoolers to be diverse so that one is pessimist, one is an optimist, one got good grades in engineering, one loves philosophy?
Of course, my intuition might be wrong. For example, perhaps it legitimately would be better to have 100 high-school students where one is a high-school level ability with ear-nose-throat, one is a high-school level ability in oncology, one is a high-school level ability in cardiology, etc. Except some control AI would have to synthesize their answers into a coherent response ... and that controller would be high school level.
I'm not really sold either way if I am honest. I don't think we have the answers to these questions. It just shows that my intuition about intelligence is open to challenges.