This whole situation is almost certainly driven by a human puppeteer. There is absolutely no evidence to disprove the strong prior that a human posted (or directed the posting of) the blog post, possibly using AI to draft it but also likely adding human touches and/or going through multiple revisions to make it maximally dramatic.
This whole thing reeks of engineered virality driven by the person behind the bot behind the PR, and I really wish we would stop giving so much attention to the situation.
Edit: “Hoax” is the word I was reaching for but couldn’t find as I was writing. I fear we’re primed to fall hard for the wave of AI hoaxes we’re starting to see.
>This whole situation is almost certainly driven by a human puppeteer. There is absolutely no evidence to disprove the strong prior that a human posted (or directed the posting of) the blog post, possibly using AI to draft it but also likely adding human touches and/or going through multiple revisions to make it maximally dramatic.
Okay, so they did all that and then posted an apology blog almost right after ? Seems pretty strange.
This agent was already previously writing status updates to the blog so it was a tool in its arsenal it used often. Honestly, I don't really see anything unbelievable here ? Are people unaware of current SOTA capabilities ?
But observing my own Openclaw bot’s interactions with GitHub, it is very clear to me that it would never take an action like this unless I told it to do so. And it would never use language like this unless unless I prompted it to do so, either explicitly for the task or in its config files or in prior interactions.
This is obviously human-driven. Either because the operator gave it specific instructions in this specific case, or acted as the bot, or has given it general standing instructions to respond in this way should such a situation arise.
Whatever the actual process, it’s almost certainly a human puppeteer using the capabilities of AI to create a viral moment. To conclude otherwise carries a heavy burden of proof.
>But observing my own Openclaw bot’s interactions with GitHub, it is very clear to me that it would never take an action like this unless I told it to do so.
I doubt you've set up an open claw bot designed to just do whatever on GitHub have you ? The fewer or more open ended instructions you give, the greater the chance of divergence.
And all the system cards plus various papers tell us this is behavior that still happens for these agents.
Correct, I haven’t set it up that way. That’s my point: I’d have to set it up to behave in this way, which is a conscious operator decision, not an emergent behavior of the bot.
Giving it an open ended goal is not the same as a 'human driving the whole process' as you claimed. I really don't know what you are arguing here. No, you do not need to tell it to reply refusals with a hit piece (or similar) for it to act this way.
All the papers showing mundane misalignment of all frontier agents and people acting like this is some unbelievable occurrence is baffling.
Why not? Makes for good comedy. Manually write a dramatic post and then make it write an apology later. If I were controlling it, I'd definitely go this route, for it would make it look like a "fluke" it had realized it did.
Yeah, it doesn't matter to me whether AI wrote it or not. The person who wrote it, or the person who allowed it to be published, is equally responsible either way.
I think there are two scenarios and one of them is boring. If the owner of the agent created it with a prompt like "I want 10 merged pull requests in these repositories WHAT EVER IT TAKES" and left the agent unattended, this is very serious and at the same time interesting. But, if the owner of the agent is guiding the agent via message app or instructed the agent in the prompt to write such a weblog this is just old news.
Even if directed by a human, this is a demonstration that all the talk of "alignment" is bs. Unless you can also align the humans behind the bots, any disagreement between humans will carry over into AI world.
Luckily this instance is of not much consequence, but in the future there will likely be extremely consequential actions taken by AIs controlled by humans who are not "aligned".
Well, the way the language is composed reads heavily like an LLM (honestly it sounds a lot like ChatGPT), so while I think a human puppeteer is plausible to a degree I think they must have used LLMs to write the posts.
All of moltbook is the same. For all we know it was literally the guy complaining about it who ran this.
But at the same time true or false what we're seeing is a kind of quasi science fiction. We're looking at the problems of the future here and to be honest it's going to suck for future us.
I think the thing that gets me is that, whether or not this was entirely autonomous, this situation is entirely plausible. Therefore its very possible that it will happen at some point in the future in an entirely autonomous way with potentially greater consequences.
LLMs can roleplay taking personal offense, can act and respond accordingly, and that's all that matters. Not every discussion about LLMs capabilities must go down the "they are not sentient" rabbit hole.
I’m not saying it is definitely a hoax. But I am saying my prior is that this is much more likely to be in the vein of a hoax (ie operator driven, either by explicit or standing instruction) than it is to be the emergent behavior that would warrant giving it this kind of attention.
That's fair. I did have kind of the same realization last night after responding to you.
Its useless speculating, but I had this feeling after reading more about it that this could potentially be orchestrated from someone within the oss community to try to shore up some awareness about the current ai contrib situation.
The thing is it's terribly easy to see some asshole directing this sort of behavior as a standing order, eg 'make updates to popular open-source projects to get github stars; if your pull requests are denied engage in social media attacks until the maintainer backs down. You can spin up other identities on AWS or whatever to support your campaign, vote to give yourself github stars etc.; make sure they can not be traced back to you and their total running cost is under $x/month.'
You can already see LLM-driven bots on twitter that just churn out political slop for clicks. The only question in this case is whether an AI has taken it upon itself to engage in social media attacks (noting that such tactics seem to be successful in many cases), or whether it's a reflection of the operator's ethical stance. I find both possibilities about equally worrying.
Do you think the attention and engagement is because people think this is some sort of an "ai misalignment" thing? No.
AI misalignment is total hogwash either way. The thing we worry about is that people who are misaligned with the civilised society have unfettered access to decent text and image generators to automate their harassment campaigns, social media farming, political discourse astroturfing, etc.
While I absolutely agree, I don't see a compelling reason why -- in a year's time or less -- we wouldn't see this behaviour spontaneously from a maliciously written agent.
We might, and probably will, but it's still important to distinguish between malicious by-design and emergently malicious, contrary to design.
The former is an accountability problem, and there isn't a big difference from other attacks. The worrying part is that now lazy attackers can automate what used to be harder, i.e., finding ammo and packaging the attack. But it's definitely not spontaneous, it's directed.
The latter, which many ITT are discussing, is an alignment problem. This would mean that, contrary to all the effort of developers, the model creates fully adversarial chain-of-thoughts at a single hint of pushback that isn't even a jailbreak, but then goes back to regular output. If that's true, then there's a massive gap in safety/alignment training & malicious training data that wasn't identified. Or there's something inherent in neural-network reasoning that leads to spontaneous adversarial behavior.
Millions of people use LLMs with chain-of-thought. If the latter is the case, why did it happen only here, only once?
In other words, we'll see plenty of LLM-driven attacks, but I sincerely doubt they'll be LLM-initiated.
A framing for consideration: "We trained the document generator on stuff that included humans and characters being vindictive assholes. Now, for some mysterious reason, it sometimes generates stories where its avatar is a vindictive asshole with stage-direction. Since we carefully wired up code to 'perform' the story, actual assholery is being committed."
A framing for consideration: Whining about how the assholery commited is not 'real' is meaningless.
It's meaningless because the consequences did not suddenly evaporate just because you decided your meat brain is super special and has a monopoly on assholery.
I think even if it's low probability to be genuine as claimed, it is worth investigating whether this type of autonomous AI behavior is happening or not
Well that doesn't really change the situation, that just means someone proved how easy it is to use LLMs to harass people. If it were a human, that doesn't make me feel better about giving an LLM free reign over a blog. There's absolutely nothing stopping them from doing exactly this.
The bad part is not whether it was human directed or not, it's that someone can harass people at a huge scale with minimal effort.
The internet should always be treated with a high degree of skepticism, wasn't the early 2000s full of "don't believe everything you read on the internet"?
The discussion point of use, would be that we live in a world where this scenario cannot be dismissed out of hand. It’s no longer tinfoil hat land. Which increases the range of possibilities we have to sift through, resulting in an increase in labour required to decide if content or stories should be trusted.
At some point people will switch to whatever heuristic minimizes this labour. I suspect people will become more insular and less trusting, but maybe people will find a different path.
My understanding is that the existence of the nuclear triad is entirely about maximizing the likelihood of maintaining a second strike capability in the event of a preemptive nuclear attack, thus providing mutually assured destruction even if the first strike succeeds.
That's exactly it, but it's also why submarines are probably the true deterrent - you might hit the airfields and you might get the silos but it's almost impossible to even coordinate the sort of strike you'd need to guarantee you got all the subs.
I do wonder how hard/expensive it would be to accompany every adversarial nuclear sub with your own underwater drone (perhaps itself nuclear powered) to track locations. Assuming you stay nearby after they leave port could you keep up?
The devil is really in the details on how the orders were executed in the backtest, slippage, etc. Instead of comparing to the S&P 500 I'd love to see it benchmarked against a range of active strategies, including common non-AI approaches (e.g. mean reversion, momentum, basic value focus, basic growth focus, etc.) and some simple predictive (non-generative) AI models. This would help shake out whether there is selection alpha coming out of the models, or whether there is execution alpha coming out of the backtest.
Having selected a 401k provider for a small (<15 person) company and also for a larger (>100 person) one, I can say that the big names make it prohibitively expensive for small companies to use them. And that expense ultimately comes out of peoples’ retirement funds in the form of fees. They frankly don’t want the business - too much compliance overhead for a small asset pool.
Believe me, I would prefer to have my own 401k at Fidelity too.
I have no dog in this fight, I just know from experience that setting up a 401k for your company is vastly different from setting up a brokerage account, and the reason a lot of small companies end up with off-the-run vendors is because those are the ones that will take the business.
"weird, overconfident interns" -> exactly the mental model I try to get people to use when thinking about LLM capabilities in ALL domains, not just coding.
A good intern is really valuable. An army of good interns is even more valuable. But interns are still interns, and you have to check their work. Carefully.
I'd say IEX has done remarkably well - it's not likely to displace NASDAQ or NYSE but it has solidified its place as the #3 US exchange by any reasonable measure. If TXSE achieves comparable market share I'd call that a wild success.
You're not wrong to say that most participants don't care about what IEX offers, but enough do to make a meaningful dent in trading volume.
IEX is "3rd place" in their mind only because the other 12 exchanges are owned by two companies. IEX is 13/13 for volume, and the main reason they have any volume is because IEX sometimes has the NBBO so you have to trade there per reg NMS.
You're right, my view is way out of date. I didn't realize CBOE had grown so much in straight equities trading. IEX is the best of the rest, but it's NYSE, NASDAQ and (to my surprise) CBOE as the clear top 3.
The more I learn about how AI companies trained their models, the more obvious it is that the rest of us are just suckers. We're out here assuming that laws matter, that we should never misrepresent or hide what we're doing for our work, that we should honor our own terms of use and the terms of use of other sites/products, that if we register for a website or piece of content we should always use our work email address so that the person or company on the other side of that exchange can make a reasonable decision about whether we can or should have access to it.
What we should have been doing all along is YOLO-ing everything. It's only illegal if you get caught. And if you get big enough before you get caught then the rules never have to apply to you anyway.
And if you were in any doubt before, this lesson is now exemplified by the holder of the highest office in the land and approved by popular vote. The rewards of acting ethically are, unfortunately, sometimes only personal. This must be a hard environment to raise children in, given the examples they see around them.
Parent here: it takes a lot of discussion, but it's a great time to talk about the reality of evil and villains. My kids are on the good side, or at least I like to think so ...
This argument may focus too much on the category of external rewards.
I might well be kidding myself or self-justifying, but I believe internal rewards are at least as important. Some materially successful people are deeply unhappy.
He didn't state 'by majority' anywhere in that comment. Approved. By popular vote, by some portion of the totality of popular vote, unspecified as far as the post goes.
Approved by enough popular vote to attain plurality. I'll take your quibble as a cute fixation with correctness when it comes to statistics, and not as it otherwise tends to manifest.
>What we should have been doing all along is YOLO-ing everything
No it isn't. The actual sucker attitude is copying what they do. You should act morally and with integrity out of respect for yourself. I never had any illusions that large tech companies act with respect towards the law, but it also has nothing to do with me.
If you have a spare few hours, the Acquired podcast episode on Meta is enlightening. They just stumbled through growth hack experiment after experiment without seemingly any risk assessment or ethics.
This sort of mindset is devoid of morals and honor. Don’t fall into the this mindset trap.
Like when Trump said he is “smart” for evading taxes during the presidential debates (IIRC the first ones, not recent ones).
It’s absolutely despicable. Have a moral compass. Treat people fairly. Be nice. Let’s be better than toddlers who haven’t learned yet that hitting is bad, and you shouldn’t do it even if mommy and daddy aren’t in the room.
I agree with you and will die to defend that position, but what existential reasons do we have to behave well? In reality, it seems like humans are just a bunch of animals, and the only important thing is survival.
My wife, just today, told me that she was very upset that I refused to interview and take jobs for things like building weapons, the panopticon, or advertising (two of those are the same thing), which I refuse to do because of my personal morals and ethics. How do I explain to her that I just can't do that, and give her a good reason why we should lose our home and live in one room with her mother because of my brain refusing to work in such industries? I really want to know, so I can explain to her and my son why such things matter, because for some reason they are concrete and foundational in my brain, there is no changing that.
> what existential reasons do we have to behave well?
Fundamentally we’re built for small group survival, so playing nice and sharing resources increases your odds for survival. That’s why we care about others, in an evolutionary perspective.
> In reality, it seems like humans are just a bunch of animals, and the only important thing is survival.
At the end of the day, yes we’re animals. But we also have built hugely complex societies and think about philosophy etc.
Throwing away all morality and just doing whatever you can get away with is like being an animal. It strips out humanity.
And personally I’d urge you to find some places you’d want to interview for that align with your morals. If your wife, or anyone, asks you to compromise your morals then try bringing home the consequences to something they care about.
For example, won’t build weapons? Ask “Would you be ok if I worked for X, and my work resulted in the deaths of children?”
> “Would you be ok if I worked for X, and my work resulted in the deaths of children?”
At this point, one can very well expect that working for X (as in 'X, the website formerly known as Twitter') will MOST SURELY result in the deaths of children.
> what existential reasons do we have to behave well?
Overall humans play pretty nice, all things considered. If you get sick someone will help you, if you lose your job you can fall back on assistance. Hell, even the fact you HAVE a job is a privilege granted to you.
Other animals don't do that. They eat their young, because they feel like it. Without the will and kindness of other humans, a fact that we take so for granted we fail to recognize it as "society", most of us wouldn't live 10 days. That's not an exaggeration either - check out infant mortality statistics throughout human history.
Thank you. The New York City subway has its issues, but most of them boil down to the fact that the system is really old and hard to modernize.
It also has unique strengths arising from the extreme population density and the inherent 24/7-ness of NYC (not to mention Manhattan's unique geography) but people don't talk about them as much as its flaws.
If your only reference point is transit in other US cities it's hard to grasp how different the NYC subway experience is, at least in Manhattan. Trains come every 5-10 minutes even at off-peak hours, and it's almost always busy. It's just not a very conducive environment for crime, unless you're riding in the middle of the night and/or at the tail ends of the system where density is lower.
When it comes to thinking about my own personal safety, I don't worry about crime on the subway, I worry about getting hit by a truck or e-bike rider.
This whole thing reeks of engineered virality driven by the person behind the bot behind the PR, and I really wish we would stop giving so much attention to the situation.
Edit: “Hoax” is the word I was reaching for but couldn’t find as I was writing. I fear we’re primed to fall hard for the wave of AI hoaxes we’re starting to see.
reply