Any resources you'd recommend for learning about reasoning about the security of code?

For low-level system security, I'm a fan of https://llsoftsec.github.io/llsoftsecbook/LLSoftSecBook.pdf as an overview for any systems developer, not just compiler devs. It might not be the most approachable, but it's got great info on everything memory corruption.

> friendly Socratic arguing with another engineer who happens to be a robot

Ha! Same! Still feels like the best way to go about it, really. I know the dream is to one day remove humans from the loop... but I'll enjoy the dialectic while it still seems the most productive!

Same, I like to call it rubber duck coding (now the duck talks back!)

Edit: Now I want an LLM connected rubber duck with a speaker/microphone that sees your screen

Reminds of me of RubberDuckGPT (rubber-duck-gpt.com):

“I won't give you answers. Instead, I'll reflect your questions back to help you think more deeply about your problems.”

Totally doable and I would buy one. Only problem is that most of the time when I'm doing "SWE" stuff I'm around other people and can't have the conversation out loud.

For my own projects, I'm very happy with an outcome that is "not faster, but better" as a result of my use of generative AI.

I still hope this will be a shared goal in at least some tech companies long-term. But the headwinds are strong. "Not better, but faster" is starting to look like a job requirement.

> as long as humans keep producing new content and innovation

Well.. we won't have to as we'll have models to do it for us!

I'm sure you were kidding, but seriously, the fact that AI-produced music pretty much all sounds the same is a good indicator that AI isn't particularly creative.

It’s not about creativity. The incentive to produce drops to zero when an LLM is just going to slurp it up and regurgitate it without some form of compensation (notoriety, money, whatever).

goddam, that's beautiful. thanks for sharing!

This is unfortunate, because (and probably much to the article writer's chagrin), I am a fan of making use of the formatting tools available for a given platform. When convenient, I love using an em/en-dash. And, when I'm up for it, italicizing and bolding words to convey prosody. It just feels right.

Now, though, I may be dismissed as an LLM (probably as an older model that can't quite use them formatting tools effectively).

I've been called out more than once for using too much italics in my writing.

But the trick is I usually write like I would speak. This leads to italicizing any word or phrase I'd speak emphatically. (Which, yes, I've also been called out for doing a lot when I speak. So what; I've also been told I'm good at getting my point across. I'll take it!) In any text important enough to go through multiple revisions, or to be written from the start with multiple revisions in mind, this characteristic is diminished. But most text is more throwaway, just like most speech, so it gets left a little rough.

This also tends to feel pretty natural. If you read LLM-written text out loud, or the prose TFA is talking about, it... does not feel natural at all. So what I'm trying to say is: some level of emphasis is just fine. Don't overthink it.

I'm speaking more of excessive headings for small sections, which are themselves just bullet point lists, all of this highly nested. That's the main thing demonstrated in the post here.

The LLM writing is basically infodump. Lists dressed up in different ways (sections, bullet points, nested bullet points, and sequences of "[bold]Blah blah[/bold]: blah blah blah" sentences.

Yea, probably. My "old-man" pet peeve is people using "lol"/"lmao"/etc. as punctuation lol.

As this post has been (to my sensibilities) obviously composed by an LLM, I can tell you: this does not read "human."

"AI use detection" is, like any test, not without cost. Meaning that, as a teacher, accusing a student of using an LLM, it may be prudent to consider the cost of a "false positive" accusation. I've seen a couple of examples now where students find sudden spurts of motivation and show unexpected talent on an assignment, to be accused of AI use after handing it in.

One should ask oneself: How many insults to the intelligence and creativity of an unexpectedly excelling student (that hasn't used AI) is it worth catching the shortcut-taking, LLM-using student? Is it 1/10? 1/1000? How much "demotivation of an unexpectedly excelling student" is the "rightful punishment of the cheating LLM using student" worth? And what is the exact cost of a false negative (letting the LLM using student off the hook)?

In other words, where on the Receiver Operating Characteristic (ROC) curve do you want to sit, as a teacher? I imagine it's quite the dilemma.

~30 years ago I sat down with two students and accused them of copying each others’ work, because they both made the same amusing mistake: they called their C functions without passing arguments, but they declared their variables in such a way that the values would coincidentally be in the right place on the stack. I have to imagine debugging their own code was a mystery.

They indicated that while they worked closely together while learning the material, they weren’t stealing from each other. I believed them then, and still believe them now, but I’m so glad I don’t have to deal with today’s AI world.

The non-LLM version of this happened to me in a high school English class, back in the 2010's. I was accused of turning in a downloaded story from the internet, but in reality, I had pushed past the incredible barrier I usually felt at the start of a task, got into the flow, and started enjoying it.

I'm not sure if it had any lasting effects. Maybe a burning hatred of Grammarly ads.

>To intentionally misspell a word makes me [sic], but it must be done.

LLM killed traditional poetry, what you are now seeing is post-LLM poetry.

Maybe you missed it, but this is clearly not an LLM, what prompt would even produce that.

Honestly, I don't think llms killed poetry.

The anti LLM crowd killed poetry by accusing every text on the net to be LLM generated and establishing generated text to be a mortal sin.

Neither of those two things would be very bad if people weren't so damn zealous about it. Nobody expected the Human Inquisition

I can already see it playing out. Some day (maybe soon) LLMs will come up with such quirks; I (and perhaps you?) will continue to insist that this does not make them "conscious" or "AGI" or "persons" or what-have-you; and I will be accused of goalpost-shifting.

See: https://en.wikipedia.org/wiki/Jevons_paradox

Worst time to be an employee, as you are expected to work faster and faster. (The approach is very much quantity over quality.)

Best time to be a solo founder in underserved markets :)

From where did you hear this rumor?

Was a news article from a reputable source (Reuters or PBS?), around the time of the Greenland flap in early January, but Google Search now sucks and the results are all polluted with news articles from the strikes today so I wasn't able to find it again.