Unfortunately, to a majority of the population approximately 100% of LLM output seems entirely legitimate.

I agree wholeheartedly, and this is the core problem - many of the people evangelizing LLMs for a particular task (especially investors and AI gold rush "entrepreneurs") do not have enough expertise in that particular field to effectively evaluate the quality of the output. It sure looks the part though, and for those with a shallow understanding, it is often enough.

That, combined with the confidence any of its output is communicated back to the user.

I’ve been trying ChatGPT for transit directions on Shanghai’s metro and it has been absolutely terrible. Hallucinating connections and routes.

But all of it’s responses definitely seem convincing (as it has been trained to do)

Except for things they happen to know something about.

Unfortunately, too few people are making the obvious leap from "LLMs aren't great for topics I have expertise in" to "maybe that means LLMs aren't actually great for the other topics either."

We as humans aren't good at it. Before AI it was already coined as the "Gell-Mann Amnesia" effect

There's another term for this that I think should catch on: Cargo Culting

Everything looks right but misses the underlying details that actually matter.

There is a larger problem that I think we like to pretend that everything is so simple you don't need expertise. This is especially bad in our CS communities where there's a tendency of thinking intelligence in one domain cleanly transfers to others. In this respect I generally advise people not to first ask LLMs what they don't know but what they are experts in. That way they can properly evaluate their responses. Least we all fall for Murry Gelmann amnesia lol

https://en.wikipedia.org/wiki/Cargo_cult

The other fundamental problem is that to a grifter, it's not a fundamental problem for the output to be plausible but often wrong. Plausible is all they need.

That's an important one. Another fundamental problem with plausible output is that it makes a manager, or a junior, or some other unsophisticated end user think the technology is almost there, and a reliably correct version is just around the corner.

Indeed. The elderly in my family are seeing a substantial uptick of AI generated stuff that looks extremely plausible. Fortunately they're old but not stupid, so far nobody has fallen for any of these but I have to admit: they look good enough to pass a first casual inspection.

"Form without substance" is quite possibly the most accurate way to describe LLMs in general. Extremely powerful (and extremely useful) pattern finders and replicators... and little else.

its certianly why they pair well with tool calls - substance without form

It's also a fundamental problem of security research. Lot's of irrelevant, highly contextual "vulnerabilities", submitted to farm internet points (driven by a broken cve system). AI only amplifies this.

No, it is the problem of any ceremonial barrier in existence. If substance wasn’t required in the first place, people were faking it already.

A parallel to AI-slop has existed for generations now out here in meatspace: Administrative/legal people on the periphery of a technical field (though possibly alas, at the top of the org's command chain) who do not at all understand what technical terms signify, but having seen hundreds of sentences produced by real experts, become able to themselves string together plausible-looking assertions.

We call these people "compliance flies".

Any large enough organization gathers them en mass to cloud real development work with "compliance."

But the problem is fundamentally slop, and grifters. It is possible to generate valid security bug reports with an AI agent, and there certainly is value in that. I'd even say a lot of value.

But of course producing fake ones is far easier and cheaper.