Now factor in number of copies sold, distribution costs, additional revenue sources...

That's a great example of how dangerous actions are perceived as innocent. The entire model of approving specific commands is absolutely bonkers.

npm run build = run an arbitrary shell command written in package.json

Meanwhile the agent could have done any of the following without approval:

- edited `package.json` to contain any arbitrary build command

- planted malicious code in `build.js` (called by `npm run build`)

- planted malicious code in `node_modules/xyz/index.js` (imported by `build.js`)

Yup. The most secure computer is one encased in concrete and dropped into the ocean.

Concrete alone isn't enough, you also need to have it be enclosed in a Faraday Cage.

that's a great point, and also the problem with relying on a human-in-the-loop to catch these kind of issues when it can be circumvented even if they were perfect

What would a better system look like?

Agents should make better use of OS sandboxing facilities with finer-grained ACLs.

Less: Do you want to run "npm run build"?

More: "npm run build" tried to read your Chrome cookie database, do you want to allow that?

Some agents like Codex use sandboxing on Linux/MacOS but the permissions are far too coarse - they'll run the command in a relatively strict sandbox and when it fails they'll ask you to allowlist the command as a whole, forever. There should be a new permission prompt every time a command tries to do something new.

Claude suggests (or used to suggest - it's been a while) to allowlist "bash" which completely defeats the point. If you do that the agent can run `bash -c "echo literally anything"`

Don’t rely on your non deterministic agent and its creators to secure your software. Design defense in depth and trust guardrails that don’t expect Anthropic to vibe good security into existence.

If you start by treating any autonomous actor in your system as an actor with the potential to go rogue the design starts to create itself

Not using agents at all. It could edit your code to do something malicious when you run it. Not even once. Not even if the agent has a gun to your head.

Don’t give a fancy random text generator access to your computer.

You misunderstood and it's really simple. Implying that on-device scanning makes it impossible for them to access any information is misleading. Just drop that faulty reasoning because it creates a dangerous misunderstanding of how technology works.

To illustrate: Because I wrote this comment, the sun is going to rise again tomorrow.

> All the LLMs have ingested nearly every cookbook ever made, across multiple languages.

That's not a positive thing, good recipe developers are Rare. For every recipe that's been meticulously tested and documented there are 1000 that haven't been. Many cookbooks are riddled with errors.

Sure, but most recipe books are just copies of other good recipe books. There are only so many ways to bake cookies.

I've always been a pretty good cook, but I've been able to pull off some really cool stuff with the help of ChatGPT lately. It is probably just an incremental lift, and I still catch it making errors from time to time, but it has been a huge help in the kitchen.

Come on now how bad could it be? Wisdom of crowds and all that...

https://www.bbc.com/news/articles/cd11gzejgz4o

Oh. Oh no.

When people say that you "can't do" something what they actually mean is that it's completely impractical (if not impossible).

Whether something is "impractical" depends on your expectations. High-latency unattended inference is definitely viable, even though it doesn't align much with what's being run in hyperscale datacenters.

I'd like to meet the person who's been using a 1 token/second system as their primary LLM for at least a few weeks. Anyone?

I think 1 token/second is optimistic here - and even then it's over 11 days per million tokens.

You didn't leave a key somewhere, you distributed the key publicly and gave everyone permission to use it.

Now you're threatening someone for taking and posting a photo of that key.

Not only that, but you also made it so anyone using the key must also make it publicly available to others.

You can replace "linux" with "windows" in your comment and make the same point.

I've installed Ubuntu and Fedora KDE for less technical relatives and they've used them without issues.

> And before I get attacked for being a Rust shill, I meant Java :P

If all you want is C but less insane then the obvious answer here is Zig.

Zig is cool, but it is not even close to being ready for prime-time. It will be pre-1.0 for a while, and major breaking changes are still happening.

Sure, maybe don't bet your entire company on mountains of Zig code just yet, but aside from the breaking changes it's been perfectly usable and suitable for every project I've ever wanted to work on.

If all somebody want is a programming language than C/C++ on these matter, there are plentiful options of the shelf to pick from.

If all somebody want is a turn key replacement to C/C++ ecosystem, then there is nothing like that in the world that I’m aware of.

If someone is switching from C because it's too easy to trigger undefined behavior, picking one of the few other not memory safe languages is missing the point.

>> If all you want is C but less insane

Zig has orders of magnitude less undefined behavior and orders of magnitude more memory safety than C. It's not a binary situation.

Object Pascal, with 40 years of experience, no need to wait for 1.0.

> must be able to maintain equal power generation levels

This is a myth, you just need to overbuild the renewables like solar, add some storage, and then have _some_ capacity from other sources to handle the dips.