I would say the same delusion even applies to the field of machine learning in general.
The "API" of trainable algorithms is essentially "arbitrary bunch of data in -> thing I want out" and the magic algorithm in the middle will figure out the rest.
Because "thing I want" is given as a list of examples, you're not even required to come up with a clear definition of what it exactly is that you want. In fact, it's a major "selling point" of the field that you don't have to.
But all of that creates the illusion that machine learning / "AI" would be able to generate a robust algorithm for any correspondence, as long as you can package up a trainset with enough examples and shore up enough "compute" to do the number crunching. Predict intelligence from passport photos? Or chances of campaign success from political speeches? No problem! Economic outlook from tea leaves? Sure thing!
The setup will not even tell you if your idea just needs more tweaks or fundamentally can't work. In both cases, all you get is a less-than-ideal number in your chosen evaluation metric.
The process is definitely vulnerable to magical thinking.
I think it is possible to avoid, though, by asking if humans can be generally good at the task in question, if working through the implied interface restrictions, and then evaluating whether the required skills can be reflected in an available training data set.
If either of those cannot be definitively answered, it’s probably not going to work.
An interesting example here is the failure of self driving vehicles based on image sensors.
My take is that most of the problems are because a significant fraction of the actual required training data is poorly represented in data that can be collected from driving experiences alone.
As in: If you want a car to be able to drive safely around humans, you need to understand a lot about what humans do and think about. - then apply that same requirement to everything else that occasionally appears in the operational environment.
To understand some traffic management strategies expressed in infrastructure, you’ll need to understand, to some degree, the goals of the traffic management strategy, aka “what were they thinking when they made this intersection?”.
It’s not all stuff you can magically gather from dashcams.
Yeah, my understanding was also that the (remaining) hard part of self-driving cars is guessing the intentions of other traffic participants. There are a lot of assumptions human drivers can make about pedestrians, e.g. whether a pedestrian has seen the car or not, whether they will wait for it, have no intention of crossing at all - or will just run across the street.
A model might potentially be able to understand those situations, but it would need a lot of highly task specific training data and it would never be clear if the training really covered all possible situations.
The other problem I see is that a lot of situations in traffic are really two-way communication, even if it is nonverbal and sometimes so implicit we don't realize it. But usually pedestrians will also try to infer what the driver is thinking whether he saw them, etc.
In those situations, a self-driving car is simply a fundamentally different kind of traffic participant and pedestrians will interact with it differently than they would with a normal car. That problem is independent of machine learning and seems much harder to solve to me.
That's not the "API" that's powered the AI boom though. What you're talking about is supervised learning. Generative AI is mostly unsupervised. It's "bunch of data -> similar data conditioned on some input". This goalless nature is one of its strengths.
The sort of questions you're talking about are primarily popular in academia. Run some MLRs against some random dataset you found, publish a paper, maybe do a press release and sell a story to some gullible journalist. It doesn't have huge value. But generative AI isn't like that.
The "API" of trainable algorithms is essentially "arbitrary bunch of data in -> thing I want out" and the magic algorithm in the middle will figure out the rest.
Because "thing I want" is given as a list of examples, you're not even required to come up with a clear definition of what it exactly is that you want. In fact, it's a major "selling point" of the field that you don't have to.
But all of that creates the illusion that machine learning / "AI" would be able to generate a robust algorithm for any correspondence, as long as you can package up a trainset with enough examples and shore up enough "compute" to do the number crunching. Predict intelligence from passport photos? Or chances of campaign success from political speeches? No problem! Economic outlook from tea leaves? Sure thing!
The setup will not even tell you if your idea just needs more tweaks or fundamentally can't work. In both cases, all you get is a less-than-ideal number in your chosen evaluation metric.