Are you sure?
I've been confidently wrong about stuff before. Embarrassing, but it happens..
And I've been working with many people who are sometimes wrong about stuff too.
With LLMs you call that "hallucinating" and with people we just call it "lapse in memory", "error in judgment", or "being distracted", or plain "a mistake".
Yeah, for the most part. But I've even had a few instance in which someone was very sure about something and still wrong. Usually not about APIs but rather about stuff that is more work to verify or not quite as timeless.
Cache optimization issue or suitability of certain algorithms for some problems even.
The world is changing a lot and sometimes people don't notice and stick to stuff that was state-of-the-art a decade ago.
But I think the point of the article is that you should have measure in place which make hallucinations not matter because it will be noticed in CI and tests.
For the most part, yes. Because people usually read docs and test it on their own.
But I remember a few people long ago telling me confidently how to do this or that in e.g. "git" only to find out during testing that it didn't quite work like that.
Or telling me about how some subsystem could be tested. When it didn't work like that at all.
Because they operated from memory instead of checking. Or confused one tool/system for another.
LLMs can and should verify their assumptions too. The blog article is about that. That should keep most hallucinations and mistakes people make from doing any real harm.
If you let an LLM do that it won't be much of a problem either.
I usually link an LLM to an online source for an API I want to use or tell it just look it up so it is less likely to make such mistakes. It helps.
I do agree. I still think that the article articulates a very interesting thought... the better the input for a problem, the better the output. This applies both to LLMs but also for colleagues.
It's the tragedy of the commons all over again.
You can see it in action everywhere people or communities should cooperate for the common good but don’t. Because many either fear being taken advantage of or quietly try to exploit the situation for their own gain.
The tragedy of the commons is actually something else. The problem there comes from one of two things.
The first is that you have a shared finite resource, the classic example being a field for grazing which can only support so many cattle. Everyone then has the incentive to graze their cattle there and over-graze the field until it's a barren cloud of dust because you might as well get what you can before it's gone. But that doesn't apply to software because it's not a finite resource. "He who lights his taper at mine, receives light without darkening me."
The second is that you're trying to produce an infinite resource, and then everybody wants somebody else to do it. This is the one that nominally applies to software, but only if you weren't already doing it for yourself! If you can justify the effort based only on your own usage then you don't lose anything by letting everyone else use it, and moreover you have something to gain, both because it builds goodwill and encourages reciprocity, and because most software has a network effect so you're better off if other people are using the same version you are. It also makes it so the effort you have to justify is only making some incremental improvement(s) to existing code instead of having to start from scratch or perpetually pay the ongoing maintenance costs of a private fork.
This is especially true if your company's business involves interacting with anything that even vaguely resembles a consolidated market, e.g. if your business is selling or leasing any kind of hardware. Because then you're in "Commoditize Your Complement" territory where you want the software to be a zero-margin fungible commodity instead of a consolidated market and you'd otherwise have a proprietary software company like Microsoft or Oracle extracting fees from you or competing with your hardware offering for the customer's finite total spend.
About 7 or 8 years ago I worked at a startup which got money from Softbank / Masayoshi Son. Our founder and our CTO went to meet him in LA IIRC to pitch.
They came back telling us he was basically asleep during the pitch meeting which was scheduled for only 10 minutes anyway.
Our business/product really had no chance of succeeding at this point and most knew it. We got some money from Softbank anyway - forgot how much. Our management was basically laughing about how easy it was to get funding from Softbank.
I jumped ship a year later or so and that was good timing.
I think it's mostly the fact that C dependencies are much rarer and much harder to add and maintain.
The average C project has at most a handful of other C dependencies. The average Rust, Go or NodeJS project? A couple hundred.
Ironically, because dependency management is so easy in modern languages, people started adding a lot of dependencies everywhere.
Need a leftpad? Just add one line in some yaml file or an "Alt-Enter" in an IDE. Done.
In C? That is a lot more work. If you do that, you do it for advanced for stuff you absolutely need for your project. Because it is not easy.
In all likelihood you write that stuff yourself.
CVE-2024-3094 is it? You can argue that in C it is much easier to obfuscate your exploit. Implementing something in C is also a lot more work, so you might be also inclined to use 3rd party library.
I never found it hard to add a C library to my projects using pkg-config. And yes, when the package came from Debian I have some trust that it is not a huge supply chain risk.
I think the problem started with the idea over language-level managers that are just github collections instead of curated distribution-level package managers. So my response "C has no good package manager" is: It should not have a packager manager and Cargo or npm or the countless Python managers should all not exist either.
Usually the hard bit with C libraries is having dependencies with dependencies all of which use their own complex build systems, a mix of Make, CMake, Autotools, Ninja, etc.
Then within that for e.g. a mix of using normal standard names for build parameters and not e.g. PROJECTNAME_COMPILER instead of CMAKE_C_COMPILER
The package manager takes care of the dependencies. And one does not need to compile the libraries one uses, so how complicated this is does not matter. I install the -dev package and I am done. This works beautifully and where it does not the right move would be to fix this.
I think in most of my projects, many of the C++ packages I used for work (lots of computer vision, video codecs etc) I had to compile and host myself.
The latest and greatest of OpenCV, dlib or e.g. gstreamer weren't available on the distros I was using (Ubuntu, Fedora, Centos). They'd lag a year or more behind sometimes.
Some stuff was outright not available at all via package manager - in any version.
So, yes, you do have to figure out how to build and package these things by yourself very often.
There are also no "leftpad" or similar packages in C. If you don't want to write something yourself.
In constrast - virtually every software package of any version is available to you in cargo or npm.
Virtually every package is in cargo and npm because there is no curation. This is exactly why it is a supply-chain risk. The fix is to have a curated list of packages, but this is what Linux distribution are. There is no easy way out.
Also pretty sure it is a feature because the general population wants to have pleasant interactions with their ChatGPT and OpenAI's user feedback research will have told them this helps.
I know some non-developer type people which mostly talk to ChatGPT about stuff like
- how to cope with the sadness of losing their cat
- ranting about the annoying habits of their friends
- finding all the nice places to eat in a city
etc.
They do not want that "robot" personality and they are the majority.
I also recall reading a while back that it's also a dopamine trigger. If you make people feel better using your app, they keep coming back for another fix. At least until they realize the hollow nature of the affirmations and start getting negative feelings about it. Such a fine line.
When you are commenting your schema, that's true.
Anything which is generated by machines doesn't need comments either.
But when it's written by people? And the values? That belongs with the 'payload'.
The wealth gap widening is quite independent from AI being involved. A natural progression which was always happening and continues to be happening. Entil some sort of catastrophe reshuffles the cards. Usually a war or revolution. The poor simply rising up or a lazy and corrupt ruling class depriving their country of enough resources and will to defend itself that some outside power can take it.
John has had meaningful impact to computing from algos, operating systems, programming languages, frameworks, etc.
Fowler has written *nothing* anybody ever cared for. Nothing.
Ousterhout has taught Software Design at Stanford where he's had students implement the most diverse software and finding the limits of his own theories and having them questioned rigorously year after year.
I always tell my engineers to create atomic commits and we usually review commit by commit.
Obviously commits like "fixed review comments" or "removed some left-over comments" or "fixed typo" should not be pushed into a PR you asked others to review.
I expect people to understand how to clean their commit history - if they don't I teach them.
The senior people who are capable of structured work - e.g. are used to contribute open source projects - do it anyway. Because messiness is usually not tolerated by maintainers of important projects.
You find people how aren't able to craft clean commits and PRs usually thrive in environments in which people are either work mostly alone or in which cooperation is enforced by external circumstances (like being in the same team in a company). As soon as developers many are free to choose whom to associate with and whose code they accept - rules are usually made and enforced.
The reason for that tidiness with some open source projects is because they want to conserve the valuable time of the maintainers, and are willing to expend a lot of time from people wanting changes to do that.
That's not the situation in a normal corporate environment. You want to reduce total time expended (or total cost, at least). It's going to be cheaper to just have a chat with your coworker when a PR is confusing.
> Obviously commits like "fixed review comments" or "removed some left-over comments" or "fixed typo" should not be pushed into a PR you asked others to review.
Could you explain this a bit more? I'm having trouble visualizing the end to end process.
1. Someone has what they feel is a complete change and submits a PR for review.
2. The reviewers read part of it, first half looks good, and halfway through they have concerns and request changes.
3. The submitter now has to fix those concerns. If they are not allowed to push an additional commit to do this, how do you propose they accomplish this? Certainly they should not force push a public branch, right? That would cause pain to any reviewer who fetched their changes, and also break the features on GitHub such as them marking which files they have already read in the UI. But if we cannot push new commits and we cannot edit the existing commits, what is the method you are suggesting?
Everytime you push commits to a PR you decide what you push exactly.
The first time you push, you should have squashed/rebased your changes into a structure that make sense. Atomic commits are best. Could even be a single commit. Sometimes it makes sense to have multiple commits. E.g.
- introducing a new API
- moving other code to use the new API
- deleting the old API
This could also be a single commit. That is really up to you/your team.
And yes, you rebase/squash and force push new commits.
Every team I had in the past 12 years routinely used force-push for PR iterations.
Turns out, when writing production code, other people rarely checkout and work simultaneously on branches of half-finished stuff of other people. It is and should be very, very rare.
Very occasionally it happens that someone bases their work on the branch another developer. In these cases, people just carefully rebase their branch on "origin/other-branch" after a fetch. You can't rely on people not force push anyway. Even if you agreed on it, sometimes this needs to be done. (e.g. commit a very large binary file by accident). So you need work in a way which assumes that somebody might have force-pushed their private branch.
Multiple people working on the same branch without a PR process is indeed messy and you should never force push when you do that. They key here is to avoid working with multiple people on the same branch in the first place.
I've seen this happening only when:
- Work items are too big and not broken down enough (branches are actively developed for for several weeks/months ). Usually and indication of lack of architecture and product leadership. If you do this, you have lots of other interesting problems as well. You are prototyping really but pretending you don't.
- You are consciously experimenting and prototyping. Make whatever mess you want - in code and history. You are going to iterate so much and so messy that whatever you produce can't a product. Figure out what you want and need to do and start with a clean implementation afterwards. And maybe delete that messy branch eventually.
So, we have two modes:
- Prototyping - which means you are allowed to make a mess because you throw it away anyway. No one, including you, cares much what you code and your repo history looks like.
- Production - you write code and repo history for eternity. You do it right for the sake of everybody's sanity.
