These are the actual real dangers of advanced AI, and they could potentially predate any AGI/ASI problems down the road, even if it's for months, the technology close the AGI state-of-the-art level would be advanced stuff, really powerful stuff, codebreaking stuff, revolutionary military strategy making machines, total global supply of stuff absorbing systems (due to fully automated industry production, 24x7x365), capable of taking out of business even the most cash-backed current players (because of trade markets predicting very low future revenues due to impossible lower prices of the automated competition), etc.

We may probably not survive the very first wave of really advanced AI technology or we could get out in very bad shape, and then we could have AGI right there, or worst.

But we could already be quite shaken. Interesting time are coming.

>the data is the limit

Most people don't know exactly how the dataset is "fed" to the training pipeline, but with the current state-of-the-art you can say the feeding is like when a human reads aloud a text, not re-reading ever, not a single word.

And then you're asked about what words you've read most often, their order and how many times they appeared in the text. Then, with those numbers you just gave, some probabilities are calculated and anotated, and there you get a "token".

There are obvious improvements plausible to be applied to that basic processing, and most are being applied already, but there's plenty of room for evern further improvement apparently.

Claude says that the previous text could be described like "a simplified metaphor of model training", so you're warned about simplicity.

You build this system, a LLM, it uses a technical process that it outputs seemingly the same output that you - a human - could output, given a prompt.

You can do "reasoning" about a topic, the LLM can produce a very similar output to what you could, how do you name the output of the LLM?

Birds can fly, they do by a natural process. A plane also can fly, we do not "see" any difference between both things when we look at them flying in the air, far from the ground.

This is mostly it about LLM "doing reasoning" or not. Semantics. The output is same.

You could just name it otherwise, but it would be still the same output.

> Being profitable is probably a matter of time and technology maturing. Think about the first Iphone, Windows 95, LCD/LEDs, etc.

Juicero, tulips....

> then you "just" assemble an AI team, and start training bigger, better LLMs, ASAP, AFAP.

There's a limit to LLMs and we may have reached it. Both physical: there is not enough capacity in the world to train bigger models. And data-related: once you've gobbled up most of internet, movies and visual arts, there's an upper limit on how much better these models can become.

> There is no "superintelligence" or "AGI"

There is intelligence. The LLM current state-of-the-art technology produces output analog to natural intelligences.

This things are already intelligent.

Saying that LLMs aren't producing "intelligence" is like saying planes actually don't fly because they are not flapping their wings like birds.

If you run fast enough, you'll end flying at some point.

Maybe "intelligence" is just enough statistics and pattern prediction, till the point you just say "this thing is intelligent".

> There is intelligence.

There isn't

> Maybe "intelligence" is just enough statistics and pattern prediction, till the point you just say "this thing is intelligent".

Even the most stupid people can usually ask questions and correct their answers. LLMs are incapable of that. They can regurgitate data and spew a lot of generated bullshit, some of which is correct. Doesn't make them intelligent.

Here's a prime example that appeared in my feed today: https://x.com/darthsidius1985/status/1802423010886058254 And all the things wrong with it: https://x.com/yvanspijk/status/1802468042858737972 and https://x.com/yvanspijk/status/1802468708193124571

Intelligent it is not

> Even the most stupid people can usually ask questions and correct their answers. LLMs are incapable of that. They can regurgitate data and spew a lot of generated bullshit, some of which is correct. Doesn't make them intelligent.

The way the current interface for most models works can result in this kind of output, the quality of the output - not even in the latests models - doesn't necessarily reflects the fidelity of the world model inside the LLM nor the level of insight it can have about a given topic ("what is the etymology of the word cat").

The current usual approach is "one shot", you've got one shot at the prompt, then return your output, no seconds thoughts allowed, no recursion at all. I think this could be a trade-off to get the cheapest most feasible good answer, mostly because the models get to output reasonably good answers most of the time. But then you get a percentage of hallucinations and made up stuff.

That kind of output could be - in a lab - fully absent actually. Did you you notice that the prompt interfaces never gives and empty or half-empty answer? "I don't know", "I don't know for sure", "I kinda know, but it's probably a bit shaky answer", or "I could answer this, but I'd need to google some additional data before", etc.

There's another one, almost never, you get to be asked back by the model, but the models can actually chat with you about complex topics related to your prompt. It's obvious when you're chatting with some chatbot, but not that obvious when you're asking it for a given answer for a complex topic.

In a lab, with recursion enabled, the models could get the true answers probably most of the time, including the fabulous "I don't know". And they could get the chance to ask back as an allowed answer, asking for additional human input, relaying on a live RHLF right there (it's quite technically feasible to achieve, not economically sound if you have a public prompt GUI facing the whole planet inputs).

but it wouldn't make much economic sense to make public a prompt interface like that.

I think it could also have a really heavy impact in the public opinion if they get to see a model that never makes a mistake, because it can answer "I don't know" or can ask you back to get some extra details about your prompt, so there you have another reason to do not make prompts that way.

> The current usual approach is "one shot", you've got one shot at the prompt, then return your output, no seconds thoughts allowed, no recursion at all.

We've had the models for a while and still no one has shown this mythical lab where this regurgitation machine reasons about things and makes no mistakes.

Moreover, since it already has so much knowledge stored, why does it still hallucinate even in specific cases where the answer is known, such as the case I linked?

>We've had the models for a while and still no one has shown this mythical lab where this regurgitation machine reasons about things and makes no mistakes.

It would be a good experiment to interact with the unfiltered, not-yet-RHLFed interfaces provided to the initial trainers (nigerian folks/gals?).

Or maybe the - lightly filtered - interfaces used privately in demos for CEOs.

So the claim that LLMs are intelligent is predicated on the belief that there are labs running unfiltered output and that there are some secret demos only CEOs see.

Another explanation for the "bubble" is that there is sufficiently convincing confidential information spread among key players in the world economy, pointing to amazing, not yet publicly disclosed LLM technology running behind firewalls and NDAs, highly advanced beyond the known public "bleeding edge" AI developments.

Hence, some potential achievements, maybe AGI, ASI, are actually atenable, because the predicted path to achieve them is some more cash dumped into a giant hardware backend, from where you, eventually,

would get to retrieve enough intelligent entities that will be capable of designing a better AI, or will be able to solve any social-organizational/technologic/economic bottleneck found in the way, thus enabling a simpler and/or easier achievement of the gotten objective, being AGI, ASI, whatever.

Allowing mankind to get there going through a better path.

if it's energy, a new source, a revamping of previously known source (micro-nuclear reactors everywhere, ubiquitous solar power, freeing the current fossil generated electrical networks thus allowing a much greater world provision of energy)

if it's economic growth without humans, because of a shrinking population, it could be technology automation (such as prompt based helpers everywhere, heck, the prompt technology could be just an offspring of the actual original - still secret - idea of a powerful LLM, to enable the next step in the AI evolution path).

Maybe the AI bubble is not a bubble, but just the beginning.

Remember that the current models, if proven as the definitive path to AGI/ASI will be with us for centuries to come, probably enhanced, they would be effectively inmortal.

This AI technology, a software that can effectively code another software is different from previous kinds of software, it's a new paradigm in itself. Maybe somewhat similar to what happened when Visual Basic first appeared (propelling hundreds of thousands of applications for the Windows environment in just a couple of years).

The first thing you'd want to achieve is to make work an AI agent capable of writing software, good software, then you could just ask it to make other types of agents, QA soft engineers, SRE engineers, DBA engineers, what ever you need.

Devin for starters is mentioned capable of training and tunning models. So, there you go.

Call me crazy, but I think that an internal monologue is a "you" instance running beside your main instance.

The "you" is because "you" are a lot of smaller stuff instances of things running along in your brain.

Probably it's common to develop this multiple "yous" and kept them happily running (may be with therapy some get to "shutdown" or "suspend" some "evil" "you" they have running and messing everything).

But some people don't need several instances of themselves, and they just run one "you" thing. Hence, no internal monologue.

Anytime now.