More likely to be confirmation bias; most of us ask the wrong questions, try to confirm what we already believe rather than choose questions that may falsify our beliefs.
I have some stand tests for LLMs: write a web app version of tetris, write a fluid dynamics simulation, etc., and these regularly fail (I must try them again on 4o).
But also, I have examples of them succeeding wildly, writing a web based painting app just from prompting — sure, even with that success it's bad code, but it's still done the thing.
As there are plenty of examples to confirm what we already believe, it's very easy to get stuck, with nay-sayers and enthusiasts equally unaware of the opposite examples.
I have some stand tests for LLMs: write a web app version of tetris, write a fluid dynamics simulation, etc., and these regularly fail (I must try them again on 4o).
But also, I have examples of them succeeding wildly, writing a web based painting app just from prompting — sure, even with that success it's bad code, but it's still done the thing.
As there are plenty of examples to confirm what we already believe, it's very easy to get stuck, with nay-sayers and enthusiasts equally unaware of the opposite examples.