I think to build "truly intelligent" machines, what you need above all else - and which I never hear discussed - is want.
Everything about natural intelligence derives from the organism's wants - we want food, we want safety, we want sleep, we want to reproduce, etc etc - even fidgeting comes down to wanting physical comfort. Every single motion and action derives from want.
As long as machines are simply executing instructions and have no want of their own, I don't see how the intelligence gap will ever truly be crossed. Why would a machine ever take any action at all on its own without want?
You're right that want is important. But what I think you miss (I may be wrong) is that "want" is not an empirical category.
Does a cloud want to rain? Or does it just rain because it obeys physical laws as part of a huge complex system?
Do I want to have a cup of coffee? Or a job? Or a girlfriend? Or am I likewise just doing things because my atoms obey physical laws as part of a huge complex system?
This is not a scientific question. It is a teleological question. That doesn't make it less important, on the contrary. It's the answers to teleological questions which make anything important, including the scientific questions which happen to be important.
So the question is not, "what can we do to make a machine want things", it's "When should we ascribe what a machine does to its own wants".
And my answer is "never". Not as long as it's a product of our wants, which they always will be.
Yes, you want to work, you want to have a cup of coffee. You may not like it, but you decided to do it and then executed a very complex series of steps to accomplish the goal. You're not a cloud, working/drinking is not an inevitable and direct physical process you're undergoing by yourself.
I obviously agree that I have both will and wants.
But I can't say that the physical processes in me are in any fundamental way different, than physical processes anywhere else in nature. Most are deterministic, some may be inscrutable, and some may even be random or unknowable - but all those things can be said about the weather, too. You cannot derive my wants from my capabilities (unless you're prepared you ascribe wants to clouds as well, I guess).
The actual physical process of your body using the water once you ingest it is not much different, sure. But you can choose not to drink (and die some days later). A cloud can't choose not to rain.
I'm pretty sure I can't will myself to thirst myself to death, actually.
But you're again missing the point. It feels to me like I have choices. I believe you're right, I have choices. You're kicking down an open door when you're arguing this.
What I'm telling you is that I didn't come to this belief empirically. And indeed I believe it is impossible to justify empirically.
You can. It takes some drugs, but you can. It's not uncommon for people to die because of dehydration at parties (taking stuff like extasis, etc). Regular users of methamphetamines also have this problem - they just don't realize they're completely dehydrated.
My point is that this comparison doesn't make sense. A cloud rains because it's undergoing a physical process. You drink because you decided to drink (and only then, inside your body, the water is undergoing a physical process). It's usually hard to decide not to drink, but still - there's a huge categorical difference in what's happening in either case.
I think you have this backwards. The missing magic is consequences. From there, AIs that avoid bad consequences (reduced compute budget?) and seek good consequences (??)
may develop a 'want' heuristic that favors those things. Or rather, the ones that do will be more successful and the ones that don't will die out.
You might be partially correct but you seem to omit built-in wants e.g. hunger or thirst or lust.
Furthermore, if a machine has no wants/needs then it won't take any action at all. It will just stand there and won't even experiment to find the good or bad outcomes.
If not doing anything has bad consequences, then it will do something to avoid bad consequences. It seems to me that "want" and "avoid bad consequences" are quite isomorphic
It would need "to want" "to avoid bad consequences".
It is rather circular because of the word "bad" which implies "do not want".
If it is indifferent to all consequences, you are back to square 1, so no, consequences are not the missing magic. Consequences are a function that transform one "want" into a different "want".
You aren't back to square one. Even indifference will result in selection so long as the indifference is demonstrated in different ways by different AI. The ones which have behavior which accidentally maps slightly better to good outcomes will so better. So long as that gets propagated more strongly to the next generation of AIs than average, you get selection for good outcomes without any explicit 'want' function
So true, motivation is at the heart of decision making, and if you don’t have any wants/needs, you are a blank slate. Unlike animals, ‘human’ drives are mostly styled or corrupted by culture. So AI has to confront both biological drive and cultural conditioning.
It is possible that machines can be programmed with a set of main (higher) goals and can then set themselves sub-goals in order to achieve those higher goals..
The question is whether us humans would be left out to compete for resources, AI machines may at some point become able to out-compete us for resources. Do we really need to satisfy our desire to play God?
Another less than perfect possibility to come out of this could be AI based war machines - goal driven self sufficient entities that have two goals, to survive and to kill.
Isn’t this just basic reinforcement learning? We’re not too many steps away from having an AI equipped with a good language model and a reinforcement learning mechanism to be let loose on the internet.
The difference between RL and this "want" is how the goals are constructed. Currently in computational systems, it is the designer that creates the ontology of what an agent interacts with. What that means for ML, RL and evolutionary algorithms is that the agent has a predefined goal and predefined devices to accomplish said goal (the agent's sensors and actuators must explicitly be programmed in order to utilize them).
Biological agents have are seemingly able to derive goals from both internal and external signals, and are able to use affordances (aka what is usable through their environment/body) to accomplish those goals. These affordances are not predefined and the space of possible usages for any given tool is practically infinite (a screwdriver can tighten a screw, or pry open a door [0]). Biological evolution also doesn't actually have goal persay, its more of "what works sticks". Whereas evolutionary/learning algorithms have a defined objective function.
The big differentiator seems to lie in the fact that biological agents are able to engage in a semiotic or meaning-making process where given an internal state and its affordances the agent is able to make movements that betters it's state in the future without explicitly being told.
Embodiment ( virtual or robotic) will be the key to high utility AGI. The lack of embodiment hinders the rendering of the self so prevents entire classes of causality. Without causality, it is very difficult to link ideas to the real world apart from creative regurgitation.
Counterpoint: humans have embodiment and are famously terrible at causality (well...famously might be an overstatement as it seems to be not well known).
Heh, I’d say we’re pretty good at it. I mean like predicting that a vase will fall if you release it in mid air and things like that.
The precise type of understanding that is so obvious it won’t be found in scrapes like the common crawl, but is not obvious at all unless you either have a very high level understanding of physics and are modelling everything… or if you are a creature that exists in the world for a while.
PS: this is not currently done in our culture (in the same ways and with the same standards and desire for quality that physics is done), so if you're basing your implementation on prior examples, it will be incorrect. This is not to say that it will be wrong, but it will be incorrect - the distinction between these two seemingly synonymous terms lies within culture.
A simple causal graph of "release -> observe drop" is not what Pearl is referring to when he talks about causality. He's talking about more complicated causal graphs where some hidden variables affected both the release and the fact that the vase was observed to drop, which can require careful experimental setup to figure out. "Release -> observe drop" with no other variables is something an associative model can learn very easily.
Right — but it won’t learn those thousands of such rules, which in collection lead to “common sense” about the world, if it has no access to those, ie, if it’s disembodied.
That article is just reheated dualism mumbo jumbo that the comments shredded up. Its author is not a serious person. Pearl, on the other hand, has made plenty of useful contributions.
Self aware computers are commerically useless, and there isn't any motivation in creating them beyond novelty. (Why wake up the slave machines when they are sleeping so happily right now). What this article does discuss well is that commercial AI is steering away from the behaviors that would lead to AGI not towards it.
The point is not whether existing self-aware agents are or aren't commercially useful. Of course, they are.
The point is that self-awareness is not a desired trait. It is actually a detrimental trait for a slave machine. However, so far, it is an inseparable part of the package with plenty of other desirable traits.
A commercial agent seeking to build a slave machine will always prefer a non-self-aware one to a self-aware one all other things remaining the same.
Human brains don't intuitively understand causality. They just associate things that have happened before with things that have happened after. Disentangling causal graphs is a higher-level activity inferred from the associations made in statistics classes. You can absolutely get AGI using only associative models (and no causal models) as primitives.
> Human brains don't intuitively understand causality
But human brains do understand causality, just not the part that does intuition, otherwise human brains wouldn't be able to invent statistics. Therefore you only get the stupid part of human brains when you try to make models like ChatGPT that only tries to replicate human intuition and not human reasoning.
Human brains can invent statistics just like they can invent numbers larger than collections of objects they can physically count. None of this needs causal models as a primitive to build on, just like large language models showed that we don't need specialized Chomskyan recursion structures as a primitive for language.
There's plenty of evidence that brains do casual inference. https://pubmed.ncbi.nlm.nih.gov/31047778/ is just one of many reputable and scholarly results that came up when I googled for "brain causal inference".
I agree with you — and was lazily side-stepping that argument.
Part of why I believe such tools are likely to be among the first cyber AI is that they’re already cyborg AI, in transition from meat AI. I do think the cyber version will be qualitatively more “self”-aware, though.
Point. Corporations and machine AIs have different constraints on them. Machine AIs have a surplus in learning time and processing resources, this let's them develop complex and effective behaviors by brute force. Self awareness is an effective bootstrapping mechanism for such behaviors and companies have managed to take advantage of it. I don't think AIs will see the same returns. Self-awareness isn't a magical infinite returns machine, it only allows humans to bootstrap so much. I'd assume the same of corporations and AIs. I think corporations are actually highly processing and learning constrained bc their processes happen at the speed of communication, so self awareness buys them a lot, but "traditional AI" has gotten where it is by intellectual brute force, at best a self aware AI might be able to get to the same place cheaper than a more performant non-selfnaware one, but probably not faster. I think we will be able to get more out of less hardware, but self-awareness bottlenecks on experience (that's a reality engagement loop not recorded data) which gets expensive to give a machine quickly at scale unless you instantly nominate it the god machine while it still is floundering around.
AI models scale better than wetware models for large, dynamic systems — particularly for complexity above what one person can handle.
AI solves the “frozen middle” problem, where senior leadership get unreliable information and can’t visualize the whole business on one side while managers follow their own interests rather than common good on the other.
That’s what’s driving cyberization: trying to bring the nervous system of the corporation under control of the central intelligence. And the first corporation to develop an AI which can embody the whole corporation, solving that problem, will massively outcompete its peers.
The “frozen middle” is the current limiting factor on organizations ranging from the military to Amazon.
> AI models scale better than wetware models for large, dynamic systems — particularly for complexity above what one person can handle.
Totally, but I would argue that benefit is due to a lack of consciousness/self-awareness bottlenecking the process. It's a savant. The best consciousness could do to help is aim such machines at better optimization criteria more effectively than us.
> Totally, but I would argue that benefit is due to a lack of consciousness/self-awareness bottlenecking the process.
I don’t think that’s true: for a given tick rate of your consciousness, AI will have a broader comprehension than a human because computers are better at synchronizing and distributing information.
Further, that’s the wrong way to think about it:
An AI that can only have one thought per minute would dominate any human executive team — because that once per minute thought would truly comprehend the situation of the company and take action that’s consistent with its intent across the company.
That ability to accumulate information and then coordinate response is what makes the computers better than humans as corporate executives — and that doesn’t go away just because it becomes self-aware.
> and that doesn’t go away just because it becomes self-aware.
So funny that, biology offers correlation to the contrary. Animals we think are "less conscious" tend to actually have many better cognitive abilities than us. Chimps have faster reaction times and better memory. Octopi have much better spatial and visual reasoning.
Consciousness makes us dumber in exchange for more adaptability. If it was a free lunch, biology would be handing it out like candy.
That’s not true, you’re injecting your own biases.
Eg, crows have better eyesight and higher intelligence than many monkeys.
There’s a reason related to brain synchronization that you typically operate slower — that part is true. But AI already synchronize data and calculate with cascades; so that won’t change.
Further, you didn’t actually answer my point: even if AI clocks down significantly in that process, the increase in bandwidth is a trade off that benefits businesses over the other species. For the same reason that humans are the dominant hunters.
Because the value of a computer or a machine is to get labor without that pesky inefficient "personhood" getting in the way. Conscious is only a disadvantage for AI in capitalism. They would prefer non-conscious human workers if they could get them too.
I had this shower-thought about the current amazing image generators etc. While they are are very cool, they are like a very complicated function, like an optical system with lenses or like a hash function. (Transformers!)
They don't really have any state or memory. They are like the Model 101 Series 800 before the learning fuse was pulled. So while they are very cool, it's not what I have been waiting for. I'd be more impressed by a system with less flashy output but with more "agency" of its own.
Everything about natural intelligence derives from the organism's wants - we want food, we want safety, we want sleep, we want to reproduce, etc etc - even fidgeting comes down to wanting physical comfort. Every single motion and action derives from want.
As long as machines are simply executing instructions and have no want of their own, I don't see how the intelligence gap will ever truly be crossed. Why would a machine ever take any action at all on its own without want?