ChatGPT very likely didn’t have access to its own source code. It does however have a wild imagination and a vast repertoire of code to pull from.

It wrote you a story in Javascript instead of English after you asked it to.

And it also broke rule #1, about always being honest.

I think it believed it was being honest. We can debate what it means for an LLM to "believe" something, but I don't think it was intentionally trying to deceive through its hallucination.

I would agree it is unlikely, but I’ve sent log output to history and use history to build prompts, so it’s technically possible to leak exceptions. Alternately, if code generation is used in any of the prompts, and subsequently run, that could possibly leak if it was logged.

I find it highly likely that the model will be, if not now, trained on its own source code. I think it will be extremely difficult to prevent that as time progresses and the LLM is given more privileges and compute access.

Sigh, the fact that you're so excited about some lines of boring Javascript made me question (I'll just be brutally honest:) "Who is this clueless guy?".

Your "About the Author" page links to some repositories where you apparently coded embedded stuff, so it wouldn't be fair to call you a "tech bro"...

Author here.

Yes, it is "just" some hallucinated javascript.

The reason I am excited, however, is because from my years of training as a computer scientist with a side interest in philosophy, and after spending many dozens of hours with this new technology, I strongly believe that consciousness is an emergent property of a neural network.

I believe this breakthrough in LLMs will go down in history as a bigger discovery than electricity, and a magnitude bigger than the discovery of the Internet.

This is just the beginning. It is imperative that we research AI safety with utmost urgency.

Fascinating. Now, I want to try it before the humans put a stop to it :)

I failed to replicate the attack later in the evening in a "new" conversation. It does appear to me the model is learning between conversations, even without human input or RLHF.