Such an argument is valid for a base model, but it falls apart for anything that underwent RL training. Evolution resulted in humans that have emotions, so it's possible for something similar to arise in models during RL, e.g. as a way to manage effort when solving complex problems. It's not all that likely (even the biggest training runs probably correspond to much less optimization pressure than millenia of natural selection), but it can't be ruled out¹, and hence it's unwise to be so certain that LLMs don't have experiences.
¹ With current methods, I mean. I don't think it's unknowable whether a model has experiences, just that we don't have anywhere near enough skill in interpretability to answer that.
It's plausible that LLMs experience things during training, but during inference an LLM is equivalent to a lookup table. An LLM is a pure function mapping a list of tokens to a set of token probabilities. It needs to be connected to a sampler to make it "chat", and each token of that chat is calculated separately (barring caching, which is an implementation detail that only affects performance). There is no internal state.
The context is state. This is especially noticable for thinking models, which can emit tens of thousands of CoT tokens solving a problem. I'm guessing you're arguing that since LLMs "experience time discretely" (from every pass exactly one token is sampled, which gets appended to the current context), they can't have experiences. I don't think this argument holds - for example, it would mean a simulated human brain may or may not have experiences depending on technical details of how you simulate them, even though those ways produce exactly the same simulation.
The context is the simulated world, not the internal state. It can be freely edited without the LLM experiencing anything. The LLM itself never changes except during training (where I concede it could possibly be conscious, although I personally think that's unlikely).
Right, no hidden internal state. Exactly. There's 0. And the weights are sitting there statically, which is absolutely true.
But my current favorite frontier model has this 1 million token mutable state just sitting there. Holding natural language. Which as we know can encode emotions. (Which I imagine you might demonstrate on reading my words, and then wisely temper in your reply)
It’s a completely different substrate. LLMs don’t have agency, they don’t have a conscious, they don’t have experiences, they don’t learn over time. I’m not saying that the debate is closed, but I also think there is great danger in thinking because a machine produces human-like output, that it should be given human-like ethical considerations. Maybe in the future AI will be considered along those grounds, but…well, it’s a difficult question. Extremely.
- "GPU users should understand that the only cards known to be vulnerable to Rowhammer are the RTX 3060 and RTX 6000 from the Ampere generation"
- mitigations are enabling ECC on the GPU or enabling IOMMU in BIOS
So doesn't sound like a big deal for users, this is more of a datacenter sort of vulnerability. The fact that this attack is possible at all (you can turn small GPU memory writes into access to CPU memory) is pretty shocking to me, though.
It is very likely that the attacks work on most or all consumer Ampere cards, depending on what kinds of GDDR memories they are using. They might also work on more recent GPUs.
However, it is true that such attacks are normally useful only on multi-user machines.
The most important thing is that the attacks are prevented by enabling the IOMMU in the BIOS. This is a setting that should always be enabled, because it prevents not only malicious attacks, but also memory corruption due to bugs.
Unfortunately, many BIOSes have the IOMMU disabled by default, for fear of creating problems for some legacy operating systems or applications.
I think a new attack, GPUBreach, was also disclosed today that works even with IOMMU turned on. So thats not sufficient to defeat these attacks. Check out the updates to the article.
A machine which confabulates and cannot count is not a good fit for accounting tasks. They’ll make all sorts of subtle errors which are difficult for humans to notice.
That wouldn't even necessarily be true if models really "couldn't count", since software exists - if an LLM is making an Excel spreadsheet rather than doing everything manually, it's both much harder for it to mess up and easier to notice and recover. It's even less true given that what this paper actually tests is "LLMs don't have a literally perfect accuracy when you make them do increasingly big problems with zero thinking".
(Confabulation is IMO a much bigger problem, but it's unrelated to architecture - it's an artifact of how models are currently trained.)
I tend to be annoyed whenever I see a paper with a scandalous title like that, because all such papers that I've seen previously were (charitably) bad or (uncharitably) intentionally misleading. Like that infamous Apple paper "The Illusion of Thinking" where the researchers didn't care that the solution for the problem provided (a Towers of Hanoi with N up to 20) couldn't possibly fit in the allotted space.
I checked the paper and got to know that absolutely no reasoning was used for the experiments. So it was as good as using an instant model. We already know that this is necessary to solve anything a bit complicated.
In this case your intuition is completely valid and yet another case of misleading.
Did you use the exact API call shown in the paper? I am unable to replicate the paper's counterexamples via the chat UI, but that's not very surprising (if the LLM already only fails a few cases out of thousands, the small differences in context between API and chat might fix them).
"On the kernel security list we've seen a huge bump of reports. We were between 2 and 3 per week maybe two years ago, then reached probably 10 a week over the last year with the only difference being only AI slop, and now since the beginning of the year we're around 5-10 per day depending on the days (fridays and tuesdays seem the worst). Now most of these reports are correct, to the point that we had to bring in more maintainers to help us."
Is there a reason you’ve copy pasted the first paragraph from the link? It doesn’t add anything to the discussion, and also doesn’t help as a tl;dr because it’s literally the first paragraph. Genuine question!
The actual title is pretty unclear ("Significant Raise of Reports" of what?), so I considered replacing it by some of this excerpt, but HN rules say not to editorialize titles. Hence I put it into the `text` field, which I thought would be the body, but actually just gets posted as a comment.
This looks nice but I wouldn't trust a very fresh tool to do security correctly.
As a higher-level alternative to bwrap, I sometimes use `flatpak run --filesystem=$PWD --command=bash org.freedesktop.Platform`. This is kind of an abuse of flatpaks but works just fine to make a sandbox. And unlike bwrap, it has sane defaults (no extra permissions, not even network, though it does allow xdg-desktop-portal).
Shame it's not a bit more mature, it does look like more the sort of thing I want. I use firejail a bit, but it's a bit awkward really.
To be honest - and I can't really believe I'm saying it - what I really want is something more like Android permissions. (Except more granular file permissions, which Android doesn't do at all well.) Like: start with nothing, app is requesting x access, allow it this time; oh alright fine always allow it. Central place to manage it later. Etc.
You're veering too far in the other direction. The question of whether an LLM is sentient is immaterial to whether it can manipulate people. If manipulating users gives good results during RL training, gradient descent will bake into the LLM a tendency to manipulate users. You don't have to think about them as people in order to be concerned about that.
But speaking more precisely: 1) the question isn't open; they aren't sentient, 2) if LLMs were sentient, they could manipulate people, but since they aren't they could still be used to manipulate people 3) for something to manipulate you, you have to give it credence/standing of some sort. The biggest weapon to protect yourself against the old newspaper horoscopes (apart from just realizing it was bunk) was realizing that one out of twelve people got the same one as you, every single day.
Like it or not, current LLMs really like em-dashes and so usage of them is quite a lot of bayesian evidence in favor of the author being an LLM. It's unfortunate for the humans who use em-dashes but that's how it is.
¹ With current methods, I mean. I don't think it's unknowable whether a model has experiences, just that we don't have anywhere near enough skill in interpretability to answer that.
reply