Except this was not the case, it had of course hallucinated what the regulation actually required (I know this because the code in question had already been reviewed by human counsel). This is (supposedly) the most bleeding-edge model available.

We use a lot of genAI to help us write code, but there is no way in the mid-term we could ever rely on these tools to actually build compliant financial products. We'd have to be totally mad. Yes, lots of Fintech companies are using these agents to accelerate, but anyone who's using them to actually ship product without a human actually digging into it is opening themselves up to a world of risk.

My guess is the model makes the same mistakes as the programmers: taking 'rules' literally, unaware of sectoral joint understanding, validated interpretations and habits. (btw. this is often on the non-tech side also a difference between regulatory and legal. The former are much more result oriented while the latter are primarily risk averse.

I think adherence to regulation and compliance is nothing to do with whether you're a SWE, a risk officer, or C-level, and everything to do with your own principles, ethics, professional attitude, and pragmatism.

1. experience, i.e. knowing why and how a rule matters (in general, but also to auditors)

2. willingness to think

If these aren't present, you get overly restrictive compliance that at the same time accomplishes nothing.

IME this is less the fault of IT and more so bad auditors that won't consider, or just don't understand, what compensating controls are. If it doesn't meet their little checklist exactly, they fail the audit.

This is such a nonsensical claim. If a company is asking someone from IT to read the regulations and implement them, then obviously you’re going to get something that conforms to the written specification they were provided.

But a company that does that is basically delegating both compliance and legal functions to IT. No sane company does that.

I was a Software Dev in a small (but fully regulated and licensed) stock exchange. We used to have guidance from legal experts, market experts, and traders, but in the last project I worked on, they just dumped 300 pages of laws and regulations on my desk and asked me what needed to be done. Why? Because the experts we used to have were either fired or left. Along with any product managers. I guess company leadership thought they were no longer needed.

Insane is right. I told them that this is not how it is supposed to work. I can't tell them what needs to be done. I am not a legal expert who can just interpret these regulations.

I was forced out of the company after that, but honestly, no one would want to work in such an environment anyway.

This actually happens scarily often, especially in smaller companies. No F500 is doing this, but there are tons of "mid market" sized non-tech companies (think 80 to 150 employees in size) that basically rely on the IT department of 1 or 2 people, or an MSSP for pretty much everything. No legal team, maybe an attorney they consult with once or twice a year if you're lucky.

regulation are written ambiguously and the specifications do not match the industry

I have even seen regulators refuse to specific legislated laws because "thats not what the government meant", giving a company the choice of following the law and being fined, or breaking the law to please the regulatory agency

One good one is that providing concrete razors for compliant and non-compliant behavior accelerates the gaming of the rules.

How to say you deal with PCI compliance without saying you steal with PCi compliance.

My experience as IT in modern banks was the opposite. The legal department were absolute assholes when it came to software features. And I'm talking absolutely 100% ok features, like paying your bills from the banking application.

The least fun, trigger happy, cover their buts people I've ever seen.

Like all they could ever say was NO. I guess they were heavily incentivized to just say NO to everything.

They are incentivized to strike the best balance of certifying (who wants to go to a place that never certifies) and validity (rubber stamp mills reflect the blame).

Yes, it is meant to be adversarial, to a point.

Later I worked in a role, attempting to achieve PCI compliance. The Auditor was a really nice guy, but there was always a short list of 10 things that he wasn't quite happy with. We kept increasing the scope of compliance to keep up with him. Everyone talked about him (Semi famous local celebrity security consultant/researcher/lecturer) and claimed that if we just stuck it out we would be super duper compliant and basically unassailable. Except that it never ended. Went 12 months with the guy. Then they just stopped paying his bills and brought in another auditing firm. Compliant immediately. You never know in a situation like that whether we were actually compliant or if there was graft. But we got there. Knowing that organisation I lean towards graft. They then failed their first audit after achieving compliance.

I have done a few PCI compliance operations since. And what I have found that you cant control for the auditor, so what good IT management does, is make every single requirement completely unassailable. If you cant write a very obvious compensating control in 5 sentences, then you just move heaven and earth to comply with the letter of the requirement (even if the project to become compliant, is itself a compensating control for a while). If you get an over achieving auditor, you wont spend 200 billable hours arguing about compensating controls. If you have a shit auditor, you know you are compliant even if they aren't being as thorough as they could possibly be. Its the only ethical way to navigate the situation.

As an enterprise architect, these are all part of the meetings you have with compliance when you are working on major projects. I have had the privilege of working with some excellent compliance officers, and they are the opposite of the nay-saying caricature that is often painted of them. I found these people to be extremely creative and helpful, working together towards solutions rather than stalling or nixing viable progress.

It doesn't feel like we're living in the same world of regulation that existed prior to DOGE.

    > I also work in finance... DOGE wiped out a large amount of the regulators in government.

However, I don't see anything about finance there. I'm confused by your comment. Can you provide more specifics?

I can't go into too much detail, but for a financial institution to offer certain financial products, you have to submit a proposal to one of the above regulatory bodies to get their approval. We were attempting to do just that and we couldn't even find the proper person at the given agency who should be receiving said proposal. It was even rumored that regulatory agency who would normally review such proposals didn't have the staff to review them. And the review would be done by an entirely different group of regulators who have not done such things historically.

Additionally, these agencies do regular exams of financial institutions to ensure they are complying with regulations and handling fraud properly. These cuts have led to those exams either not happening, or happening at a fraction of the depth they had been previously.

What a win!

I'm not implying anything else. I used your own "literal" wording to refer to the "more strict than yours" interpretation.

I suppose I should have used scare quotes around "literal".

Company politics, feudal wars, fiefdom protections, backstabbing and outright sabotaging, now there's a daily occurrence and many minions are cannon fodder in those skirmishes, but they usually stay clear of regulatory issues minefields.

If the company you work for actually had such a no-fault culture, I doubt you'd be criticizing programmers so aggressively for being sticklers, but would instead be trying to understand and account for the systemic factors (including human factors) behind their behavior.

I don't see why developers should be in trouble. Developers don't make unilateral decisions on non-trivial compliance matters. A finding of non-compliance at a financial institution would typically be the result of an investigation, a disagreement with the regulator or a court ruling. It would come years after the organisation as a whole decided to adopt the interpretation in question.

Engineers are not shielded by their implementer role if they participate in illegal activity. James Robert Liang was a rank-and-file engineer for Volkswagen and he got jailed for his role the VW emissions scandal[1].

No matter how much an enterprise architect or compliance officer promises "it'll be fine" to the developer, the developer needs documented CYA. An enlightened organization would perhaps find ways to expedite that CYA documentation rather than demonizing programmers as a class.

[1] https://apnews.com/general-news-988ea2ae45694b37b320e68cefe3...

Liang got prison time because he _did understand_ that the engine wasn't compliant with regulations and chose to build the system to falsify the emissions output during tests anyway. He was not a scapegoat.

"On 9 September 2016, James Robert Liang, a Volkswagen engineer working at Volkswagen's testing facility in Oxnard, California, admitted as part of a plea deal with the US Department of Justice that the defeat device had been purposely installed in US vehicles with the knowledge of his engineering team: 'Liang admitted that beginning in about 2006, he and his co-conspirators started to design a new "EA 189" diesel engine for sale in the United States. ... When he and his co-conspirators realized that they could not design a diesel engine that would meet the stricter US emissions standards, they designed and implemented [the defeat device] software.'" from https://en.wikipedia.org/wiki/Volkswagen_emissions_scandal

Complain about them, denigrate them, upbraid them for performing analysis outside their primary expertise, fire and replace them.... none of that changes the incentive structure that shunts people in the implementation role towards conservatism out of a perceived need for self-preservation.

"decisions which they don't understand to be compliant" = "decisions which they don't believe to be compliant"

In other words, they understand that the decisions are not compliant. There's no contradiction with what you said.

a) Engineers don't know and cannot be expected to know whether what they are being asked to implement complies with all regulations. This is completely normal.

b) Engineers know or can be expected to know based on their expertise that they are being asked to cheat. That's when they are on the hook.

VW was a case of (b). It was clear-cut criminal behaviour on a very technical level. But that's not what typically happens in financial services and many other domains.

But if your point is merely that engineers are not automatically in the clear just because someone higher up told them what to do then I agree with you.

It's nice to want things, but rules are much squishier in real life. There's rarely any truly bright line.

Is neither what I said nor believe.

There’s a reason it’s called “judgement”

My point was simply that it's easy to scoff at someone else being careful if it's their neck and not yours.

Isnt it how we make stable, deterministic and predictable system? How do you want to create one with ambiguous rules?

The problem is that sucks, even if all software engineers keep their jobs and salaries, the floor is still pulled out from under us. Imagine if a surgeons job was to supervise robot surgeons from a remote computer, or a woodworker just signs off on work before the machines do all the cutting and assembly. Sure they still have important jobs in their field but the soul & humanity of their skill is gone.

For me, AIs have actually made the job more soulful, not less. For one thing, it lets me use the part of my mind that is good at human language, not just the part of my mind that is good at software. This makes the job feel a bit less one-dimensional in terms of what parts of me are engaged while doing it. For another, I find it liberating to no longer have to think much about boilerplate code or to spend time roaming around the Internet looking up documentation of various language syntax and API details, the vast majority of which are arbitrary rather than being based on any kind of mathematical beauty. For me it makes the job more soulful that I can think of the job on a higher level instead of having to spend effort on arbitrary and tedious details.

Of course there is still the question of "will the job even exist in a few years, at least for more than a relatively small number of people?". But that's a separate question. For now at least, I am finding that for me AIs have brought a lot more soul and humanity to the job than it ever had before.

However, if I were just having to do things for the man, I might have a rather different take on all this.

Does the woodworker who shape using a handsaw use less "soul" than the one who uses a machine?

Does the musician who use a DAW and VSTs instead of analogue tape recorders create music with less "soul"?

Does the painter who buys acryllic paint instead of synthesizing their own dye from plants use less "soul"?

As technological innovation progresses, the barrier to creation falls. The process of creating something is not to be conflated with the final piece of art itself.

Compare to this to prompting an LLM: “Generate a third person where game with a view from above where you can steal cars, shoot at people, run from the police, etc.” Anybody with access to the tool can do this, and the results are just another uninspiring GTA clone that you would imagine.

The latter is more like a carpenter ordering their “work” from alibaba then it is like using a skill saw.

https://www.youtube.com/watch?v=gkqNWNLKpZg

I'm critical of many "AI" developments but I can't and don't want to argue with this. I say we still need to struggle for humanity and we do need to save our souls, but that "it's a machine" is not where the battlefield is.

To write a piece of music, you're working at so many different levels, the analytical, emotional, and structural as well as drawing on years of training and experience. When a person with little (or even just a moderate amount of) music training generates a piece of music in a couple hours, are they actually a composer? I personally would say no. I mean it does take a good ear (which is important) to use a music generator well, but still, would say they are more of an editor or an evaluator or a practical critic of music instead of a composer.

Yeah, at what point in a discipline does the increasing skill of AI overtake the need for our contributions from the working being done? In music, it's happened already. Looks like it's happening in coding too.

This isn't like the step from hand saws to power saws, and it's disingenuous to pretend like it is. This is what the startup machine has been doing to every industry... finding... "inefficiencies" and "optimizing" them.

It's not the item's soul that's at stake when you stop recognizing that, it's your own.

It's when a woodworker, musician or painter completely outsources their work and just marks what's wrong, sending those parts back. Yes, the final art piece might be the same, but the artist definitely uses less of their "soul".

Do they? I saw some crazy stat from the guy who built claude code that he was pushing hundreds of PRs a day. There's no way you can human review that much code. It's probably closer to heavily AI assisted review and planning.

I would love to be able to say I pay the same amount of attention and am just as diligent and communicate as clearly with an agent, but it wouldn't be honest: I scan agent PRs for obvious mistakes or misinterpretation of what they've implemented.

With human colleagues I usually know them and their style, their way of working, so have a better idea what to look for. You also have a genuine return on providing feedback that helps coworkers learn and improve, whereas with agents, all the feedback you write is gone when the thing gets merged (unless your org has some kind of shared memory for its agents).

I don't have the answer for what the future looks like, but I suspect agent-type-1 reviews agent-type-2 is actually where we'll end up.

When industrialization hit, we definitely lost a ton of craftsmanship and craftsman, but a standard Ikea chair is less likely to wobble than the average chair at a much better price (for a random example). Yes, we traded artistry for convenience, but what we really did was bifurcate our needs between "some place stable to sit" from "a beautiful chair for my home". Most people wanted the former more than the latter, and the same applies to software.

If we split the roles into buckets, many woodworkers disappeared, some became artisans, some became designers for industrially-produced products, and some catered to Luddites for a long transitional period. Despite Anthropic's claims, SWEs won't disappear in a year but over a generation or two, no matter how good LLMs become.

Obviously software is much more complicated and integrated into other elements of business, which in a way makes it more vulnerable to AI taking over and in another way will be at the mercy of larger shifts to how businesses organize human roles and responsibilities. What we call "taste" comes down to "intent" - what the hell does a company do? What should it be doing and how should it operate? These will be the only questions that matter and the one thing LLMs can't replace since they will always choose the most default path. So I think human's roles will be to inject intent/taste at different levels of abstraction throughout an organization.

In addition the incentives are misaligned - the "artisan" made chair (in the past) wasn't likely made for aesthetic reasons - it was made to last long term and function. And if it wobbled or had any problems the original woodworker was probably around to fix it.

I had no prior experience with woodworking, and while it's a little bit wobbly, it's much more robust and it's the exact shape and size I need. It cost the same and it'll last much longer.

And there's emotions and a story attached.

I don't believe there is any point in having ai generate code

If robots make life saving surgery cheaper so that many people can afford it, isnt that a good thing?

Here's an example of what we will continue to see with folks fully immersed in gen AI psychosis:

"The creator of claude code said that he no longer writes code for about 6 months and now has Claude doing all his work now. He also said recently that he no longer prompts Claude and now has it running in loops and it is self-improving itself and performing better than a human!"

If the code produced by the LLM is perfect, the LLM takes the credit. But when a disaster happens, you cannot blame the LLM and it then falls on the human who did it.

I don't think SWEs heavily vibe-coding with LLMs realize the risk in not understanding what the code the LLM being produced is doing even after generating tests (lol). We will see more of this too. [0]

[0] https://sketch.dev/blog/our-first-outage-from-llm-written-co...

No, thank you. I have used the new tools, determined that they aren't helpful to me, and set them aside as I would with any other bad tool. I don't feel the need to let hype take the steering wheel.

Exactly. You are free to use openclaw or a coding agent to build a competing bank, hedge-fund, hospital or even a new airliner because the previous ones were built by humans. Surely an AI can do it better by itself.

So why haven't you done it yet?

Yes, me. Yes, I tried LLMs for what I am doing and will try again in few months. No, there was no noticeable or clear improvement over doing it manually.

Yes, I am using some LLMs for some purposes but Claude Code had slight improvement, if any, not worth introducing proprietary dependency.

Because we can actually see the disjointed slop that Anthropic produces. And when issues happen, they can't fix them for weeks on end because no one understands what code does anymore, and all of their "hard problems causing issues" they blog about are literally "if we had actual engineers this wouldn't even be an issue to begin with". Like this bullshit they had in spring: https://www.anthropic.com/engineering/april-23-postmortem

> It would be like a developer in 2020 claiming that he only writes assembly because compilers can’t be trusted.

LLMs are not compilers. For a few very obvious reasons I'll leave as an exercise to figure out

If the AI is producing what you tell it to, why are you needed?

This seems like a really confident prediction. It isn't true right now, why do you think it will be true in the future? Right now having knowledge and experience is a huge benefit to steering the LLM (it makes dumb decisions all the time still).

Well I use tab completion, of course. And I copy-paste snippets from LLM more often than from SO now. But otherwise not much has changed in my career in the last 5 years. Is this different for you?

I'm not fundamentally opposed to code generation, and I use LLMs for some taks, but I don't see myself vibecoding whole pages of production code. I vibecoded a throwaway note-taking app for myself though.

These days it very much feels like the task has shifted from "building the system" to "building the system that builds the system".

It only looks to be trending one way.

I work at a big tech company and I don't know a single person that still hand writes code. Most people haven't hand written code for at least half a year now.

I do wonder what sort of bug is making its rounds on HN that people here find this so shocking and unbelievable.

The absolute vast majority of people who point out AI's downsides have used it and use it. People who uncritically write things like "I work at a big tech company and I don't know a single person that still hand writes code." scare the shit out of us for a good reason.

Any eng that is only using Claude Code or Codex or whatever, is frankly not entitled to talk about AI's limits since they are using the most basic harnesses. They literally don't know better.

When I see Claude Code or Codex users on HN talking about how coding with AI is risky, it's like watching someone that has only ever seen a catapult argue about how space travel must be impossible.

I really doubt big tech has much better harnesses than are publicly available. Definitely not "catapult vs space travel". They have the same base models we all have access to.

Harnesses aren't testing boundaries of AI. They inject "make no mistakes" in various forms and provide some session management tools.

And, as with any people fully buying into and promoting hype, "my harness is in another castle" (they never show anything they boast about") lathered with huge amounts of crude demagoguery and analogies.

I've seen accidental non-compliance. I've seen what I would call negligent compliance, where a company attempted to be compliant but didn't meet full, correct compliance (one example I've seen is that a company assigned resources to compliance and forgot to increase resources as workload increased, causing them to be increasingly behind on compliance work), but I've never seen a company that just decided to pretend to be compliant knowing that they were not.

Security, GDPR, backups, build pipelines, disaster recovery, most of it will be faked, half-heartedly done once or ignored entirely.

Then there's the more abstract things like scalability, idempotency when integrating with external APIs, error recovery, accessibility, UX, etc.

Almost always that sort of stuff will have been entirely ignored, or there will be a fig leaf over a real mess of misunderstood standards or manual intervention steps.

Startup developers usually have to be generalists as they often wear many hats, so things that need deeper domain knowledge get done to a bare minimum.

Early stage startups, maybe, but most of those are three failsons in a trenchcoat. That sort generate more PR than revenue.

We have long historical experience and innate tools for detecting and mitigating errors made by humans. If we can't apply those to automation, then even fewer total mistakes may end up being a worse outcome.

Genuine question: your top coder seems to be producing the most error-free code from your perspective, has the deepest knowledge of the architecture and codebase, and is faster on the trigger than the others.

But your top coder has proven and verifiable dementia, where they will confidently assume the existence of apis and code that do not exist, mix up the purpose of others and forget other things, and you can't predict when and how they will introduce errors into the system or the severity of such errors.

Are you really comfortable letting this person with dementia generate most of your codebase in the airline and health industry?

I also hope you have an iron-clad agreement that prevents the model provider from doing silent updates because all your evidence of correctness you collected thus far goes out the window in that case.

Another genuine question:

You have witnessed a human coder and the AI you're using make the same important mistake. Assuming you do not have the time and resources to retrain, fine tume, and test your frontier model:

Who would you trust not to make the same mistake multiple times in the future after you have warned them that their job depends on it, the AI or the human?

The parent is implying they would prefer an AI when working in the airline and health industry because it makes less errors. Read the comment again.

They have not said, "Hey, I work in the airline and health industry and I'd love to use AI for a couple of the bullshit IT UIs we have as long as we can put guardrails on the AI to stay in its lane."

I asked a yes or no question. The guardrails you can put to mitigate errors are the same guardrails pre-AI for the humans (tests, regressions, reviews). If you were wary of employing a top lead engineer with verifiable dementia prior to AI for a mission critical system, logic implies you should think twice giving that much responsibility to an AI as well.

> The hallucination thing I think is mostly overblown

Can you predict when and how the SOTA model will hallucinate? Yes or no. Can you predict the severity impact of that error beforehand? Yes or no.

>from speaking to colleagues it seems to vary wildly depending on which model and harness you are using

You have partially answered my question it would seem.

No, but the same can be said for your colleagues. You might call what the LLM does hallucinations, I'd call them mistakes. I think we have totally forgotten that humans make them all the time and are confidently wrong too.

Your original question, doesn't really get to the bottom of the point I'm trying to make, and I don't really feel it fairly represents the issue we are talking about here. They are not the same things.

Also, if a human does this, you can replace them and get a human who will not do it. The default for an LLM is to generate plausible-looking text that may or may not be completely incoherent. That is not the default for a human. Again, if you find that your colleague consistently fabricates APIs, you can hire someone who isn't crazy instead, but you cannot do the same with LLMs.

That's absolutely false. My collegues don't routinely and confidently invent apis that are not there, or spectacularly and repeatedly misunderstand the purpose of certain functions or exhibit extreme forgetfullness. Especially when I've warned them. Hallucinations and confabulations in otherwise healthy individuals are mental disorders. When I ask them why they made an certain kind of error, I can expect to get a reasonable answer. No one has uttered the phrase "Bob hallucinated again while writing those tests" when the Bob in question is a human.

Calling hallucinations simply mistakes does not seem to me to be a healthy way to reason about LLMs. I can ask a collegue how well they can program in Ada and adjust my expectations on productivity and bug rates. I can't ask an LLM how well they can code in Ada (just a throwaway example), or even how much of Ada was in its training data. I have to actually spend money and spend time code reviewing before I can even formulate any expectations at all.

And it took "convincing" that it had made a mistake.

But the most reasonable take, which I'm happy to see reflected in so many comments in this thread, is… use both.

Do an AI pass, and have humans verify, and vice versa. Let the humans drive the AI. Then the unique shortcomings of each party can be covered by the other's strengths.

It might beat an underresourced human review, on time, efficiency, cost metrics. But on the metric of accuracy, throwing unlimited humans at a problem will still beat throwing unlimited AI at it

You can do that, sure. But doing so negates any improvements in speed the LLM brought. And at that point, you may as well just do it yourself to begin with.

I use GenAI tools when coding a lot, but I do not vibe code. I go through everything it generated, and we iterate. And yes, it doesn't save me a lot of time. But what it does do is free up mental capacity in a similar manner. But instead of syntax, it's more complicated patterns. Maybe I don't remember how to stitch something together, but i know it can be done. Instead of spending the time to look it up and then code it, I just tell it to do it for me.

That's how I use the current AI, too. I never ask them to do something without specifying how it should be done. I ask questions first, use /plan to let the model ask me questions, then I let it execute the plan while reviewing the results. More and more often, I get something close enough to what I would have written. In the opposite case, I at least know exactly how to rewrite the result, if needed.

I observe the same effect as you: while it does sometimes speed up the implementation a bit, it's not very noticeable; however, it frees me from having to recall all the obscure little details up front. Instead, I can describe them, have the model implement them, and then recognize them (and refresh my memory) when reviewing. The effect is that it's easier to start a task because I don't need to prepare as much to execute it. It's especially notable on things that I haven't touched for some time. I know, more or less, how my Elixir projects are set up, but after ~2 years of not working on them, getting back into them had been a hassle - with AI, it's no longer that. I think the biggest difference comes from the AI lowering the cost of context switching for me - I used to have huge problems with that, and AI certainly helped a lot.

We could do with less speed.

Or are current AIs too similar for that to be fruitful?

regulation questions. even the simple ones, AI gets all the time wrong. it wasn't Mythos, but other models like opus.

I can adjust the view on this topic if/when we get access to mythos.

Well too bad, the problem is that they also produce things much faster than humans so errors will compound quicker.

Maybe it's just me, but it seems that companies will happily take existential risks to get a better bottom line short term. Either you're too big to fail or you've already privatised any profits and subsequent losses (due to the risks becoming manifest) are socialised. The motor industry seems to be particularly egregious in this aspect, but also the food industry, construction industry etc.

Seems to me even governments make the same choices in many ways - cut back health-care, policing, education, public transport and let the next government deal with the consequences.

Definitely - defund the police was an astonishing rallying cry that made the communities it pretended to help much more dangerous for their residents.

But the opposite is far more likely, and far more destructive long-term: it's easier to buy votes by spending more on social programmes and rack up debt and/or inflation to cover it, and then spend even more to fix that problem for enough voters that the people paying for it all can't vote it away, and the people who vote for it over the decades just don't understand why their pot feels so uncomfortably warm all of a sudden.

I find hallucinations happen for us with those models, but we've worked on baking in guardrails and fact checking against sources of truth, so that it's less of a problem.

We're engineers. We just try to engineer out the failure modes.

And this is fine. Developing new software with a really smart intern is the same, you, as an expert, need to bring your experience/expertise on the table to have everything right. Because experience needs time.

https://www.tomsguide.com/ai/ai-can-now-identify-anonymous-i...

The original Mythos release used ASan to filter false-positives so it was able to maintain a good FPR, but when Mythos moves into domains that don't have a readily available oracle to help filter hits, the result is a deluge of false bullshit.

But really that particular issue could have been solved by literally just telling it in a markdown file or instructions something like "verify all facts or compliance requirements with web search and include citations in responses".

“Verify all facts and compliance requirements” leaves enormous holes even if you assume the LLM has a concept of facts and requirements (it does not).

What facts? What requirements? For what industry? For what subset of that industry? For what country or countries that you will be doing business in? Are these current “facts” and “requirements” or is the LLM referencing a dusty article from 1992 for which the subject matter has been radically overhauled?

In my job I regularly see small but incredibly important mistakes like this lead to major issues. Some of those are human driven but increasingly the defense of the person responsible has turned into “Claude said it was fine though!”

No. This is a disasterous instruction. Not only is it vague, but it's also meaningless. When giving instructions to an LLM your prompt must be concise and exact. Tell it _exactly_ which requirements need to be followed, ideally have it write or (preferably) pass audited tests to enforce these requirements. You also need to provide it with a hard source of truth it can rely upon. Instead of saying "verify facts", you're better off by saying "... make sure [whatever you're doing] matches with data at X.Y.Z, verify by running [instruction/command/program]"

Additionally, using a specific tool does not suddenly give the model common sense enough to say “this piece of information doesn’t answer the question of whether this solution fits in this specific industry at this time in this place”.

If I plucked a random passerby and gave them the task then no, I’d find myself detailing out every specific to them.

You’re equating the LLM to the least qualified candidates. I don’t think your argument is communicating what you intended.

feel like the parent you are replying to literally views their place of work as a daycare which is very condescending

The argument is regarding LLMs and domain knowledge.

I remember hearing that 10 years ago about self-driving.

We need a lot more basic research into LLMs and also a lot cheaper hardware.

The current batch of LLMs will turn a lot of fields upside down, but not to the tune of $3tn or whatever crazy amounts are being invested right now.

And the thing he complained about is fixable with a web search, and AI does programming and office work today. So, it's already here. It's just a question of degrees.

Tesla has been a couple years away from FSD for, what, like ten years now?

If you scrape off the glitter, you'll find a lot more duct tape and wire than you think.

IME people would benefit greatly from the process, albeit tedious and time-consuming, of testing out the same prompt sequence/session with the exact same model multiple times. It becomes clear extremely quickly how capable but unreliable and inconsistent a model can be even when given the same context. If you have ever completed a long, complicated task with an agent and then lost the session and tried doing the same thing again from scratch you may have had the experience of seeing the subtle changes that come up in the model's thinking which lead it to accept or reject certain paths and ignore or incorporate prompt instructions like the one you've provided.

You sure?

Did it do the correct job once you put the regulations doc(s) in the context?

"Make it better" with no additional or reasonable previous explanation of what better might mean.

"AI will figure it out" not for pattern extraction, but for a full blown analysis with equally generic prompt all confidently stated by an executive telling people working it how it works

So the question remains if non-programmers will adapt, the LLMs will accept wider range of input styles, or .. its just another abstraction layer for devs to use.

I've observed this in the wild where someone is iterating with an LLM and giving it only negative feedback. For example responding to edits with "don't make it blue" rather than "keep the existing button shape, and change the color back to green".

The LLM doesn't really come back the way a human would and say "so what color do you want?".. it just, guesses. Now abstract that to more complex tasks.

A spec detailed enough and unambiguous enough to be translated into machine execution deterministically is called code.

Unlike a compiler, AI can build with a spec that is not detailed enough or unambiguous enough: It does so by filling in the gaps with educated guesses.

This is safe if and only if you take the time to later read the output, understand what its guesses were, and judge wether they were acceptable. No AI can do this for you because the truth lies in your original intentions, which it does not have access to.

The jury is out there on how reliable and time consuming this is vs writing the code yourself; it is not immediately obvious that is faster or requires a smaller cognitive load.

> The system shall have behavior identical to that expressed by the system created by the following source code. [add some stuff about environment to taste]

Particularly as tokenmaxxing has ended and people are being charged more economic prices. If the pricing 5-10x the way Uber,etc did on the path to profitability.. even more so.

I mean for a lot of spec code people define the API signatures and let the llm run with it which is an excellent tradeoff.

other than there are "internal micro feedback loops" during development?

Doing the above doesn't actually make the model smarter, so, if it couldn't get to correct code with fewer steps, then the light you see at the end of the tunnel is an oncoming train.

The only way to test this is to test it out, in real life. Sometimes people see results, sometimes people don't. Note that yes, I am including the entire iteration process - even after iterating, people still don't see results with AI.

I have had both positive and negative experiences with AI, over multi-week projects. But apparently on hackernews, anything positive about AI is proof that AI is superhuman and taking over, and all follies about AI are lies by stupid humans who secretly have psychological dispositions to fear AI. Sometimes the AI genuinely isn't good enough. Are we not allowed to say that now? We might not know why, but it's just the truth.

The other solution is to formally analyze the entire space of possible actions the agent can take a priori. Then yes, you can definitively say whether or not the principle breaks or not. Can you, though? Can you give a formal specification for the space of possible actions for AI and show that your loop never breaks, or breaks less than humans, or any other sensible criteria? If not, then you can't just give an abstract principle and start making inferences from that.

You're the only one coming away thinking there was a net negative experience.

The only thought-ptovoking discussion should be "why the hell do we have this stochastic parrot anywhere near out codebase"

A system which will just randomly decide to give the legal team reasons to not back you up is:

* A system whose output will get brought up in lawsuits and make legal's job harder.

* A system that will make the dev team perpetually chase its tail while it oscillates between the several different valid interpretations of the rules.

How many gaps have humans not caught?

> But if “one error is too many” is your point of view

Yes, in regulated industries "one error is too many" is the only right approach.

Yes, humans also make errors, and there you have a range of options: from tracing and finding the causes of the error (and tightening processes) to literally jailing those responsible. Your hallucination machine will happily "identify" 17 gaps, and create 34 more. And no, there are no processes to make it better. The "make no mistakes" incantation will happily be ignored for obvious reasons, regardless of how many forms of it you throw at it.

You're basically saying "we need human review for literally everything AI outputs because we have no way of saying whether anything it produces is hallucination or not. And since it produces plausible-sounding things really fast, it puts enormous burden on human reviewers".

The machine doesn't say that. It says "Here are 170 completely correct and verified results".

You have to check and verify all of those results yourself, and on any given day it can be anywhere from 0% to 100% incorrect.

> I assume you've come to this conclusion based on some reasoning, but you're not sharing it in this response AFAICT.

The reasoning comes from actually working with AI tools. And the reasoning can be seen in the actual comment this tgread started from: https://news.ycombinator.com/item?id=48434824

>In a regulated industry 90% false positive rate is indistinguishable from 100% failure rate

So defending that position on the basis of it not actually being a 90% failure rate would mean you shouldn't have taken it in the first place. The fact that the LLM lies about its failure rate is nearly irrelevant; the machine could output the literal string "The following is 90% likely to be a false positive: " followed by the LLM output.

The reasoning in the comment that started the thread is "it's annoying to redo human review". Your position as I understand it is that there is no or negative business value to a tool that spit out a list of potential issues of which 10% are real issues with your business. This is what I fail to understand. This would be an incredibly useful first step towards any audit and would save loads of money. Why not?

The conversations had already been had and the product made compliant. Mythos just pulled new rules out of its ass and of course the product wasn't compliant with those. So they do a fire drill and find that to be the case at great expense.

Yeah you can frame it as "more checking is always better" if you wanted but that's just the same old "other people's resources are valueless" slight of hand we see on everything. It probably was mostly wasteful work.

So, in this case, the LLM's behavior was equivalent to the behavior of the resistance during WWII.

I think that book should be required reading for all engineering students.

> the code in question had already been reviewed by human counsel

But how are you so sure your colleagues are not more "expert" than you? Prior LLMs there was room for very good engineers and mediocre engineers to work together in 99% of the companies out there. With LLMs, only the "best" engineers will survive, because nobody needs mediocre engineers anymore.

This being HN, I imagine every engineer reading this thinks they are in top the 10-5% of their company/city/country, and therefore they think they are not "mediocre" engineers that can get affected by the introduction of LLMs. Statistically, they are probably wrong. So, it's all about ego. Chances are you are not a rockstar and LLMs will eventually take over your job.

As usual, the only winners here are corporations and executives. Most of us are the last monkeys in the chain, and so we'll get screwed.