AI has led us into a deep spaghetti hole in one product where it was allowed free rein. But when applied to localised contexts. Sort of a class at a time it’s really excellent and productivity explodes.
I mostly use it to type out implementations of individual methods after it has suggested interfaces that I modify by hand. Then it writes the tests for me too very quickly.
As soon as you let it do more though, it will invariably tie itself into a knot - all the while confidently ascertaining that it knows what it’s doing.
On localised context stuff, yeah no. I spent a couple of hours rewriting something Claude did terribly a couple of weeks back. Sure it solved the problem, a relatively simple regression analysis, but it was so slow that it crapped out under load. Cue emergency rewrite by hand. 20s latency down to 18ms. Yeah it was that bad.
For me it's just wildly unpredictable. Sometimes it gets a small task perfectly right in one shot, sometimes it invents an absurd new way to be completely wrong.
Anyone trusting it to just "do its own thing" is out if their mind
For me I would ask it to do a simple thing and it would give me the tutorial code you could find anywhere on the Internet. Then you ask it to modify it in a way that you can't find in any example online, it will tell you it's fixed everything, but actually nothing has changed at all or it's completely broken.
I think if someone's goal was just the tutorial code, it would have been very impressive to them the AI can summon it.
It only takes a cursory knowledge of what LLMs really are to understand why recreating tutorials is easy, but making actual new stuff that is well engineered (takes way way more than "passes the test suite") is difficult.
Actual novel stuff is so far out on the long tail of iterations that it's a gamble: it might pop up in an early run, or might take 2000 prompts and $20,000 worth of tokens. And it's still not really engineered, it's 10,000 monkeys with typewriters copying random shakespeare snippets off the chalkboard. At some point you'll get all of Hamlet, but most of the time you'll get garbage, and sometimes you'll get Romeo & The Taming of The Tempest.
this is what I've been using freebie gemini chat for mostly, example code, like reminding me of c stdlib stuff, javascript, a bit of web server stuff here and there. I think it would be fun to give googles agent or cli stuff a spin but when I read up here and there about antigravity, I'm reading that people are getting their accounts shutdown for stuff I would have thought was ok, even if they paid for it (well actually as usual the actual reasons for accounts getting zapped remain unknown as is today's trend for cloud accounts).
I'm too poor for local llms, I think there might be a 2 or 4gb graphics card in one of my junk pcs but thats about it lol
I found that unpredictability to be interesting. I'm doing super simple projects with these models and a year, or even six months ago, it would give me a block of code and as soon as you ran it, it would fail. And you'd have to paste the error in and keep going until it was smoothed out.
The other day though I asked for something simple and it one-shotted the problem. To me, that's new.
I know this success was a statistical outlier, however. I grok how to use it and to not trust it. I'm just shocked so many people smart people fail to understand it.
We had this debate at my company and the end result was a ban on “random” service names.
So we ended up with “auth service” instead of something like “Galactus”. The problem of course is that “auth service” isn’t searchable in our monorepo and it was a nightmare to find or discuss any info or references to the service itself. Now imagine if docker was called “container manager”. Good luck googling that and disentangling it from all the search results.
The value of a name doesn’t come from it being self-explanatory but rather from it being a pseudo-unique identifier. The small cognitive tax of remembering it serves as a shared bookmark between people that you can refer to when discussing or speaking to others about it - whether we’re talking about docker, Linux, or another person.
It's especially fun once the service is superseded. Guess which one of these services handles MAC address allocation and storage: macaddr_db or atom-util?
Is there any real drawback to just never giving your real name or address to service providers to minimise the chance of identity theft? Most likely it’s against terms of service, but other than account suspension are you likely to suffer any legal consequences?
The only way to fix the ToS issue you raised is through regulation protecting it.
Unfortunately we're going the other direction, with efforts like verified ID gaining traction in some parts of the world.
It's ironic because in most cases anonymity (or allowing an alternate identity that has its own built-up reputation) would offer real protection, while the verification systems are arguably security theatre.
I don't care what technical genius is built into your architecture, as soon as you force a user to plug their ID information into it, they've forked over control along with any agency to protect their own safety.
The ad tech companies can associate any fake identity with your real identity. So no, there is no problem. Good thing that all ad tech companies are fully on the up-and-up and have never been compromised to spread malware.
These sort of stories seem to be dime a dozen and weirdly celebrated around HN and the software engineering community.
We’re told of the hero, who goes against their managers and executives and doesn’t deliver any stories as agreed in sprints.
We’re told of the engineer who isn’t hired by Google because he can’t invert a binary tree. Everyone else piles on and decree that, yes indeed, you cannot measure developer efficiency with a Leetcode or whiteboard problem. We’re too good for that. Another engineer chimes in: “I don’t test my candidates. The best people I worked with were hired over a beer and a chat at the local pub”
We’re told of the MBAs who destroy the organisation, by introducing evil metrics, and how that the work we do are immeasurable and that the PHBs don’t understand how great we are. 10x engineers aren’t a real thing, everyone is equally productive in our digital utopia.
Meanwhile in the real world, hordes of awful engineers deliver no story points, because they in fact, do nothing and only waste time and lowers morale.
Meanwhile in the real world, each job opportunity has thousands of applicants who can barely write a for loop. Leetcode and whiteboards filter these people out effectively every day.
Meanwhile in the real world, metrics on delivery, features and bugs drive company growth and success for those companies that employ them.
To me, all these heroes, and above process people, just strike me as difficult to work with narcissists who are poor at communication. We are not special, and we do not sit above every other department in our organisation.
> We’re told of the hero, who goes against their managers and executives and doesn’t deliver any stories
> Meanwhile in the real world, hordes of awful engineers deliver no story points
Do you think the point here is that not delivering on one specific metric is a good thing, or that not delivering one specific metric can't be assumed to be the whole picture?
It’s the latter, but my point is that’s a tired and weak argument to make.
The blog poster could’ve asked, why does the manager want me to deliver the story points? It’s because Jake is also delivering zero story points and he’s a terrible engineer and it’s a good canary metric.
Agree in 1st and 2nd “meanwhile”. On the metrics thing though I have the opposite experience - mild distraction at best, total disaster and productivity killer at worst. Most orgs couldn’t even implement metrics fully (or at all) because in reality it is always much more work than they usually anticipate. Instead they larp as google and waste your god damn time.
Having said that i do think that there are a bunch of senior folks who think they are Tim but meanwhile they’re just hiding their laziness/incompetence by getting involved in other people’s shit
People of a revolutionary (or "innovative") temperament are those who are going to say, "this system doesnt work, these processes are broken, the wrong outcomes arise" and ignore them. In doing so they just "do the right thing" in their judgement, and in so doing, develop the next iteration on the processes that others will follow.
If these innovators are operating in a niche where innovation is required, they are solving different problems than most others and have different self-defined standards ("narcissism"), and so on.
Probably many people who visit HN have this temperament, and a significant number are in niches which need to evolve this way (eg., this applies to all startups). HN is a small sample of engineers: most don't go to websites to conceptualise their own activity, reflect, etc. These are indications of people with a desire to innovate, or to solve novel problems in their profession.
If you are in a highly stable environment, with effective processes, etc. then people of this temperament can be trouble if left entirely to their own devices: good managmenet would place them in projects/areas where there is some unknown unknowns to figure out.
In many cases however, people without this temperament (say, "it works, dont break it, conservatives") find this behaviour unsettling, arrogant, disruptive, isolating -- because it is. There isn't any thing to "communicate" when you havent figured out what the solution is -- you can air your thought process every day, but that will just unsettle more people when they see how much it changes (in response to more thkning, information, etc.). And the values by which this change takes place are not conservative, they're radical and imposed by a person who sees a route out of a predicament and so on. It's quite arrogant to place yourself in that position, or think it's yours by some invisible duty that no one else has.
In any case, if you operate in this niche, esp. eg., if you're in a start up environment -- then you arent going to care a jot about this "real world". They are acting against the real world, to improve it.
Thanks for responding. I see your point, but I think it is responding to something slightly different than the point I was making.
If I may latch on to your first paragraph, my point is that we are saying this first bit “this system is broken” and are happy to throw out the baby with the bath water and tear it all apart, on flimsy evidence and generalisations.
And yes, there’s definitely something to be said about the HN crowd having a temperament toward innovation, but I don’t think that’s in any way orthogonal to my point. In fact, this community is far more rational than most others, so I would sort of expect us to rationally look at company processes too, but for some reason we seem to have a blind spot when it comes to our managers and executives and the ‘horrors and hoops’ they make us jump through every day.
>> To me, all these heroes, and above process people, just strike me as difficult to work with narcissists who are poor at communication. We are not special, and we do not sit above every other department in our organisation.
Exactly.
Looks to me like Tim was really good at hiding his incompetence behind other people's backs. Also looks like a problem of the others, particularly senior others of not telling him "Fuck off, Timmy" when he sat uninvited beside them to "pair program" together.
So on one hand, you're kinda right. HN is filled with exaggeration (imo often justified) from people venting because they have to deal with the bad parts of this system all day. That seems natural in a dev filled space.
But I don't think your comment is fair.
> We’re told of the engineer who isn’t hired by Google because he can’t invert a binary tree. Everyone else piles on and decree that, yes indeed, you cannot measure developer efficiency with a Leetcode or whiteboard problem.
Because this is a bad way to judge engineers. Or, rather, it's a great way if they don't know how to invert a binary tree. Most of the job is to figure out something you don't know yet and do it. Giving an engineer a random wikipedia page on an obscure algorithm and having them implement it is a great interview tactic. Having them regurgitate something common is bad, there will be a function for it somewhere, and you just need to call it.
> Meanwhile in the real world, hordes of awful engineers deliver no story points, because they in fact, do nothing and only waste time and lowers morale.
I agree with you on this one. Those people need to be fired. That doesn't mean story points are a good metric, often 90% of long term value can come from the kind of people who are like Tim, and losing them can destroy projects. Just because something bad is happening, it doesn't justify killing 90% of value for a team.
The only thing I've seen that works is to give team managers more discretion and rigorously fire managers who regularly create poor preforming teams (you often have to bump manager pay for this, that's fine, good managers are worth their weight in gold).
> Meanwhile in the real world, each job opportunity has thousands of applicants who can barely write a for loop. Leetcode and whiteboards filter these people out effectively every day.
You do need to filter for people that can code. That doesn't mean filtering for inverting binary trees is a good idea. Having people submit code samples that they're proud of is a much better approach for a first filter.
> Meanwhile in the real world, metrics on delivery, features and bugs drive company growth and success for those companies that employ them.
Bullshit. Basically all companies use metrics, and most companies are garbage at delivering useful software. A company being years behind and a million over budget on a software project, and eventually delivering something people don't want is so cliche that it's expected. And these companies regularly get out competed by small teams using 1% of the resources, as long as the small teams give half of a shit. In fact, if you want my metric for team success, what percentage of the team actually cares is a good one.
You're proposing a solution with a <20% success rate. Don't act like it's a gold standard that drives business value to new heights. With the system as it is today, most companies would be better off getting out of software and having a third party do it for them.
My wider point is not that the way companies are run is perfect and that we should stop the “innovators” (to quote the sibling comment). Each of these examples speak of corporate dysfunction, but we never give any weight to the constraints that force them in place. Leetcode is bad, but it’s bad in the sense that it errs too heavily on filtering out false negatives - the cheaper of the two errors. The alternative is worse.
Giving Tim the benefit of the doubt in this story, it still holds true that for every extraordinary and invisible superstar like Tim there are 99 under-performers who are indistinguishable from him.
We need to empathise with our managers and the processes in our organisations to understand their purpose and how they came to be.
We, software engineers, keep picking out singular data points of evidence to point at a flawed and unfair world, that go against our self inflated egos.
The brew guy inverting the binary tree and Tim being great, does not invalidate the practices of whiteboards and story points as a general practice.
To your final point, the best organisations that I’ve worked with used metrics in a very effective way (mostly in start ups). The worst did too. Just because some do it poorly, does not mean that it’s bad across the board.
What is tiring, is the unfair, and low expectation of the quality of evidence demanded of the anti-establishment notions in software development, before they are taken as gospel by this community.
And, in my experience, the people who are the strongest proponents of sidestepping or dismantling these processes overlap strongly with those who also do not deliver value to their teams.
> Leetcode is bad, but it’s bad in the sense that it errs too heavily on filtering out false negatives
But, it doesn't. It filters for something orthogonal to development, which is ability to obsess over clever algorithmic solutions. Ok, well my company does HackerRank instead of LeetCode, maybe LeetCode is magically better, but I'm not seeing anything that tells me someone who grinds LeetCode is actually going to be useful on my team.
Look, you want an idiot check to make sure someone is actually able to code, fine. That's probably a good idea. But the number of stories on here about people being turned away because they hadn't run into a particular tricky algorithm problem is concerning.
> Giving Tim the benefit of the doubt in this story, it still holds true that for every extraordinary and invisible superstar like Tim there are 99 under-performers who are indistinguishable from him.
But they aren't indistinguishable. The author of the blog post was perfectly able to distinguish them. That's my point. There are plenty of ways to be able to distinguish them, this metric just isn't one of them. Because it's a bad metric.
It may not be legible to the higher ups, but good lower level managers have no problem distinguishing good unconventional people, and under-performers.
> We need to empathise with our managers and the processes in our organisations to understand their purpose and how they came to be.
I do empathize with the managers, at least the lower level ones. That's why I advocated for putting them in complete control and giving them unilateral firing privileges and increasing their pay.
> the best organisations that I’ve worked with used metrics in a very effective way (mostly in start ups). The worst did too.
You're really making it sound like metrics (at least as traditionally practiced in software) are orthogonal to being a good organization. If that's true, we might want to re-think how much time we spend on them and how much money we spend capturing them.
Now, if you want to use profit, adoption, or user satisfaction as metrics, I'd love to talk about that, but I've seen nothing in my experience in the corporate world that tells me that the way we're currently using them is even net positive value.
It only appears that HackerRank/Leetcode isn’t good at filtering because you’re viewing it from your perspective, and not the perspective of the entire population that is tested. To you, the predictive power at the top tail end of the distribution is low, because you’re thinking of two strong developers Alice and Bob. Alice happens to know algorithm X and would pass the test, whereas Bob does not. But that’s not the population we’re testing. Think more along the lines of Alice and Bob and your grandmother were the test population. It’s absolutely fantastic at filtering the lower 95% of applicants because they will _never_ be able to pass. Yes, inadvertently 2.5% of “good developers” are filtered too, but that doesn’t matter to the outcome of your company. They just want someone competent, and they don’t care if it’s Alice or Bob.
The same logic sort of applies to Tim and his performance. The bias of having an imperfect metric is probably much better than the bias of letting an army of middle managers go with their cut. Besides, it doesn’t have to be a hard filtering function at this stage, but a metric to indicate that we need to look a little closer at Tim
> It’s absolutely fantastic at filtering the lower 95% of applicants because they will _never_ be able to pass.
This is the part I disagree with. It hasn't been true for years. Anyone with the free version of ChatGPT can pass a hacker rank today.
> but that doesn’t matter to the outcome of your company
It does for mine, because we've hired all of the good developers that get through the process you're describing and it isn't enough. We actively moved away from what you're describing and turned the interview into a 2-3 hour pair programming session where the person completes a mini version of a ticket.
This has much more predictive power than what you're describing.
> This is the part I disagree with. It hasn't been true for years. Anyone with the free version of ChatGPT can pass a hacker rank today.
It certainly still is true today. Anybody who is sufficiently motivated to cheat can pass it. It was true prior to ChatGPT, and it still remains true today. And yet they don’t. Most people completely fail these screens
> It does for mine, because we've hired all of the good developers that get through the process you're describing and it isn't enough.
Then your industry is atypical in the type of applicants that you are getting. So to accommodate you’ve had to increase your false positives to reduce false negatives. That’s completely fine if it’s what you need to do, but it’s not the typical experience for a tech company.
We also do a pair screen after the code test and we still reject around 80% who make it to that stage. How do you scale interviewing everyone if you don’t pre screen?
> Then your industry is atypical in the type of applicants that you are getting
Based on the quality of candidates that get through at other companies, I'm guessing our problem isn't atypical. Or at least, good devs often aren't getting through their pipelines at all. It's possible that in trying to reduce the false positive rate, they screened out all of the actual positives, but that doesn't paint a good picture of the industry.
> How do you scale interviewing everyone if you don’t pre screen?
We do pre-screen. The fact that they haven't encountered a tricky algorithm before isn't a problem. For ones where the syntax is valid, a dev at my company does a code review on it.
AI has led us into a deep spaghetti hole in one product where it was allowed free rein. But when applied to localised contexts. Sort of a class at a time it’s really excellent and productivity explodes.
I mostly use it to type out implementations of individual methods after it has suggested interfaces that I modify by hand. Then it writes the tests for me too very quickly.
As soon as you let it do more though, it will invariably tie itself into a knot - all the while confidently ascertaining that it knows what it’s doing.