It's not at "an accident" and Go didn't "somehow" fail to replace C++ at its systems programming domain. The reason why go failed to replace C and C++ is not a mystery to anyone: Mandatory GC and a rather heavyweight runtime.
When the performance overhead of having a GC is less significant than the cognitive overhead of dealing with manual memory management (or the Rust borrow checker), Go was quite successful: Command line tools and network programs.
Around the time Go was released, it was certainly touted by its creators as a "systems programming language"[1] and a "replacement for C++"[2], but re-evaluating the Go team's claims, I think they didn't quite mean in the way most of us interpreted them.
1. The Go Team members were using "systems programming language" in a very wide sense, that include everything that is not scripting or web. I hate this defintion with passion, since it relies on nothing but pure elitism ("Systems language are languages that REAL programmers uses, unlike those "Scripting Languages"). Ironically, this usage seems to originate from John Ousterhout[3], who is himself famous for designing a scripting language (Tcl).
Ousterhout's definition of "system programming language" is: Designed to write applications from scratch (not just "glue code"), performant, strongly typed, designed for building data structures and algorithms from scratch, often provide higher-level facilities such as objects and threads.
Ousterhout's definition was outdated even back in 2009, when Go was released, let alone today. Some dynamic languages (such as Python with type hints or TypeScript) are more strongly typed than C or even Java (with its type erasure). Typing is optional, but so it is in Java (Object), and C (void*, casting). When we talk about the archetypical "strongly typed" language today we would refer to Haskell or Scala rather than C. Scripting languages like Python and JavaScript were already commonly used "for writing applications from scratch" back in 2009, and far from being ill-adapted for writing data structures and algorithms from scratch, Python became the most common language that universities are using for teaching data structures and algorithms! The most popular dynamic languages nowadays (Ruby, Python, JavaScript) all have objects, and 2 out of 3 (Python and Ruby) have threads (although GIL makes using threads problematic in the mainstream runtimes). The only real differentiator that remains is raw performance.
The widely accepted definition of a "systems language" today is "a language that can be used to systems software". Systems software are either operating systems or OS-adjacent software such as device drivers, debuggers, hypervisors or even complex beasts like a web browser. The closest software that Go can claim in this category is Docker, but Docker itself is just a complex wrapper around Linux kernel features such as namespaces and cgroups. The actual containerization is done by these features which are implemented in C.
During the first years of Go, the Go language team was confronted on golang-nuts by people who wanted to use go for writing systems software and they usually evaded directly answering these questions. When pressed, they would admit that Go is not ready for writing OS kernels, at least not now[4][5][6], but GC could be disabled if you want to[7] (of course, there isn't any way to free memory then, so it's kinda moot). Eventually, the team came to a conclusion that disabling GC is not meant for production use[8][9], but that was not apparent in the early days.
Eventually the references for "systems language" disappeared from Go's official homepage and one team member (Andrew Gerrand) even admitted this branding was a mistake[10].
In hindsight, I think the main "systems programming task" that Rob Pike and other members at the Go team envisioned was the main task that Google needed: writing highly concurrent server code.
2. The Go Team members sometimes mentioned replacing C and C++, but only in the context of specific pain points that made "programming in the large" cumbersome with C++: build speed, dependency management and different programmers using different subsets. I couldn't find any claim that go was meant as a general replacement for C and C++ anywhere from the Go Team, but the media and the wider programming community generally took Go as a replacement language for C and C++.
When you read through the lines, it becomes clear that the C++ replacement angle is more about Google than it is about Go. It seems that in 2009, Google was using C++ as the primary language for writing web servers. For the rest of the industry, Java was (and perhaps still is) the most popular language for this task, with some companies opting for dynamic languages like Python, PHP and Ruby where performance allowed.
Go was a great fit for high-concurrency servers, especially back in 2009. Dynamic languages were slower and lacked native support for concurrency (if you put aside Lua, which never got popular for server programming for other reasons). Some of these languages had threads, but these were unworkable due to GIL. The closest thing was frameworks Twisted, but they were fully asynchronous and quite hard to use.
Popular static languages like Java and C# were also inconvenient, but in a different way. Both of these languages were fully capable of writing high-performance servers, but they were not properly tuned for this use case by default. The common frameworks of the day (Spring, Java EE and ASP.net) introduced copious amounts of overhead, and the GC was optimized for high throughput, but it had very bad tail latency (GC pauses) and generally required large heap sizes to be efficient. Dependency management, build and deployment was also an afterthought. Java had Maven and Ivy and .Net had NuGet (in 2010) and MSBuild, but these where quite cumbersome to use. Deployment was quite messy, with different packaging methods (multiple JAR files with classpath, WAR files, EAR files) and making sure the runtime on the server is compatible with your application. Most enthusiasts and many startups just gave up on Java entirely.
The mass migration of dynamic language programmers to Go was surprising for the Go team, but in hindsight it's pretty obvious. They were concerned about performance, but didn't feel like they had a choice: Java was just too complex and Enterprisey for them, and eeking out performance out of Java was not an task easy either. Go, on the other hand, had the simplest deployment model (a single binary), no need for fine tuning and it had a lot of built-in tooling from day one ("gofmt", "godoc", "gotest", cross compilation) and other important tools ("govet", "goprof" and "goinstall" which was later broken into "go get" and "go install") were added within one year of its initial release.
The Go team did expect server programs to be the main use for Go and this is what they were targeting at Google. They just missed that the bulk of new servers outside of Google were being written in dynamic languages or Java.
The other "surprising use" of Go was for writing command-line utilities. I'm not sure if the original Go team were thinking about that, but it is also quite obvious in hindsight. Go was just so much easier to distribute than any alternative available at the time. Scripting languages like Python, Ruby or Perl had great libraries for writing CLI programs, but distributing your program along with its dependencies and making sure the runtime and dependencies match what you needed was practically impossible without essentially packaging your app for every single OS and distro out there or relying on the user to be a to install the correct version of Python or Ruby and then use gem or pip to install your package. Java and .NET had slow start times due to their VM, so they were horrible candidates even if you'd solve the dependency issues. So the best solution was usually C or C++ with either the "./configure && ./make install" pattern or making a static binary - both solutions were quite horrible. Go was a winner again: it produced fully static binaries by default and had easy-to-use cross compilation out of the box. Even creating a native package for Linux distros was a lot easier, so all you add to do is package a static binary.
As well as the issue that, to a first approximation, nobody cares.
The correct response to the replication crisis was for every affected science to immediately stop all funding and new experiments, if not pausing the ones in progress, review the situation with a series of intense conferences, and figure out how to allocate a lot more resources to replication. If that sounds extreme, well, the response to "vast quantities of the science you're doing is effectively worse than useless and you lack the ability to tell which is which" should be extreme!
But, hey, I can put my realist hat on too. I know that was never on the table. A lesser response was all we could ever hope for.
But what we got is effectively no response. Or if you prefer, the bare minimum that can be just barely called more than nothing. Nobody cares. The science goes on being cited, both in the field and in popular magazines, because that's just so much more fun. And an indeterminate, but assuredly large percentage, of science money is worse than wasted, but used to generate false science instead. And the prestige of "science" will continue wearing away until it is all spent.
That series of articles may be the most significant thing I read this year.
I have a comment on one bit:
In his article The Perils of JavaSchools, Joel Spolsky
refers to "programmers without the part of the brain
that does pointers or recursion". In this "ha ha only
serious" comment Spolsky displays a profound intuition.
He's obviously observed his teams very carefully. My
only disagreement with him is that I know from
experience that everyone has the necessary bit of the
brain. It's just that understanding pointers and
recursion requires juxtaposition, and in most people
that mode is not accessible.
The pointer problem
I am a programmer who tried and failed to get pointers many times over the course of some seven years, and then consciously tried a new way of thinking about it that flicked a switch in my mind nearly instantly. I was aware of the widespread belief that you either did or didn't get pointers, and immediately wrote down everything I could about the situation in order to capture it. I'll describe my findings and the situation here.
Pointers are easy when you think about the program in terms of memory and what's actually going on: casting of ints. The problem for me was that my brain was filtering out the correct scenario because due to a contradition in the syntax. Let me present a simple example to use for discussion.
One of the things that kept screwing me up with pointers in C was the contradiction in the situation where you pass a pointer to a function, yet the function receiving it has a different symbol for it. Look at the example above - where we declare 'c' in the parameter list for the function 'fn' it has a star next to it. Yet when we call fn it doesn't. But ... those are the same thing. How can that be?
I found that the more I thought about the problem the less I'd understand it. I tried to get several friends to step me through it and couldn't get there. I'd actually get it for a while, and then I'd lose it. In desperation - and I'm not making this up - I worked out a series of base examples that I knew to work even though I couldn't see why, printed them and pasted the sheets into the back of a copy of _Systems Programming for SVR4_. Horrifically embarassing - a professional unix-based programmer, who hangs around real geeks, who has designed and implemented several successful applications on a variety of platforms - reduced to this. I rapidly reached the point where I could write and debug C code without my colleagues detecting the weakness through my system. A key part of it was knowing the exact point at which I had to actively stop thinking about the syntax and trust what was written on my base examples.
OK - step back a bit. There's something else that's important. I'm told that I spoke early when I was a child, and rapidly got to the point where I could hold court in a room full of adults, and talking at their level. English was my strongest subject throughout high school. Throughout my life people around me seemed to think I was very smart but I felt like a fraud because although I was qite quick in language stuff, there were some problems that kids who struggled in general got quickly and that I didn't.
Early last year I was struggling through what felt like just such a problem and noticed myself feeling at a loss without an example to leverage. I wondered why it was that I needed to learn from example, and thought it interesting that some people are able to work through things without examples, yet I seemed incapable of it. This caused me to create a theory that I had a strong skill at certain sorts of learning based on pattern matching, and that overuse of this skill had stunted other learning skills. I told myself that I'd look out for opportunities to solve problems without using this kind of approach.
The first decent example that came up over the next few days was - conveniently - the pointer problem. A friend called me up to say she had an exam the next day and needed help with C - could I help? Readers will already recognise that are few times in any lifetime when good looking, charismatic, single girls (this one is a superb singer as well) will seek you out on the strength of your reputation for coding and I replied in a steady voice and pace, "Sure - why don't you come over now?" As I hung up I thought - oh oh - back to that old pointer problem that has been haunting me since the beginning of time. But then I realised that this was the opportunity I'd been looking for to overcome a problem with non-example based learning.
I had the whole problem solved less than five minutes from putting down the phone. I dug up my copy of _The C Programming Language_ and read the section on pointers to refamiliarise myself with the problem space. As you would know - the problem in a world without pointers is that you can't change non-global variables within a function so that the change persists past the end of the function (I had always been fine with this - nothing new so far). So I moved to the next step and decided to ask myself, "how would I solve this?" Easy - pass in the memory address. Have a mechanism for referring to memory addresses in memory instead of the variables they contain. Oh, and there needs to be some syntax to be able to dereference a variable.
That was the lightening bolt moment. The problem with pointers in C is that the [star][varname] syntax means two different things, and they are actually contradictory from a certain perspective. In one context it means "declare a variable that is a pointer" and in the other it means "deference". Whenever my languages-brain bit looked at this it couldn't deal with it, recognised it as a contradictory positions and basically shut me down, but not in a way that let me work out what was going on. Have a look at it yourself from that perspective - hopefully you'll see what I always did. Now whenever I look at C code I instead just think "everything is an int and pointers are all about casting ints, and a star when you declare something is your way of telling the compiler that the type of this thing is int-pointer instead of just int". Thus I now have no difficulties at all.
When I was at uni I heard the whole thing about people not knowing C would never be Real Programmers. And I'm glad I stayed insecure about it, or I may never have conquered what is clearly a huge deficiency in my learning patterns. OK - that's enough about me. Let's instead look at the big picture. The assumption behind the claim that groking pointers is a good guide as to whether someone has what it takes is that in order to get C you need to be able to think like the computer. But could it in fact be the case that the emphasis on C in computer science has been driving people with a strength in the humanities away from programming (in preference of people who do not have a language dominance) for the last few decades, and causing bias towards people who lack whatever it was that was sitting in my brain filtering out the correct case?
Oh - the short conclusion of my story is that - girl came over, we spent all night pretending to be a C interpreter by drawing memory diagrams on my whiteboard, she moved from likely failure to a reasonable mark in the space of four hours, was thrilled after the exam the next day, and now we both live on different continents.
I think there are leaps in what I've written here. But I'm confident I'm on to something about the relationship between people who use a certain type of thinking and have trouble with pointers. It probably doesn't apply to people with a strong background in low-level programming.
Since this article uses that fluffy "tell a story" writing style, here's the facts in the article stripped out:
1. Air conditioning consumes 10% of global energy and 20% of energy used in buildings.
2. Demand for cooling is expected to increase significantly, with 2/3 of world households projected to have air conditioning by 2050.
3. Dehumidification accounts for over half of energy consumption in air conditioners in humid conditions.
4. New technologies are being developed to improve air conditioning efficiency:
a. AirJoule by Montana Technologies:
- Uses metal-organic framework material for dehumidification
- Claims to reduce energy for dehumidification by up to 90%
- Still in prototype and testing stages
b. Blue Frontier:
- Uses liquid desiccant for dehumidification
- Being installed in various locations in the US
c. IceBrick by Nostromo Energy:
- Energy storage system for large-scale cooling
- Can reduce annual cooling costs by 30% and associated emissions by up to 80%
- Only suitable for centralized cooling systems
d. Gradient:
- Window-based heat pumps with larger external units for better efficiency
- Currently costs $3,800, aiming to reduce to $1,000
5. Electrocaloric cooling is being researched as a potential future technology, potentially 20% more efficient than current methods.
6. Passive cooling measures (e.g., window shutters) should not be overlooked as they cost nothing to run.
7. Current non-centralized air conditioners operate at only about 20% of their theoretical maximum efficiency.
Alright, so first of all, the colors don't matter, they're just for contrast/legend purposes. Flame graphs come in all sorts of color schemes, not even necessarily yellow/orange/red.
I tend to look at flame graphs in terms of % of the overall process. They're good for finding that one part of a routine that is taking up a decent % of processing, and if that part of the routine is being hung up on some mundane task.
For example, if I see 3/4 stacks directly on top of each other, then I know I've got a call stack a few levels deep that is overwhelmingly waiting on some low level thing to finish. If it's something that should be really fast (like a cache lookup), then I know something really stupid is happening.
Some flame graphs will tie in e.g. network requests and DB queries as their own traces, which will also give you a clue sometimes. Like, oh, this function is waiting 10s for a query to complete? Let's see what that is actually doing, maybe we can speed it up.
I used flame graphs (among other things) this year to take a 30 minute long payroll process down to about 3 minutes. Much of this was just scanning the flame graph for things high in the call stack that were taking up noticeable % of the processing time. This is easier if you know the codebase, but for example, I could see within the "load a bunch of data" phase of our processing that there were a few tax-related things taking up most of the overall time. We managed to trace those to a few calls to a third party library that we couldn't make any faster, but we could cache the results to mitigate the issue.
Another place we found expensive queries being repeated in different functions, which was obvious because we had 2 calls to the same function from both places. We ended up just raising those shared calls up a level and patching the data directly into the two functions that needed it.
Other places were less obvious. We could see a lot of time being spent, but we couldn't tell from the flame graph what was happening. We'd go look at some code and find some n^2 aggregation function that we'd need to simplify.
Overall, flame graphs are just one tool. They might not even be the best tool. In our case (heavy data driven web application) I would place DB observability at least as high in importance as good tracing, and flame graphs are just one way of visualizing traces.
I'm lucky to have been surrounded by researchers who have been teaching and thinking about these ideas over the last ~6 months, so this essay has been marinating for a long time... your article just provided the necessary activation energy for me to write this all out. Thank you!
I'm always deeply impressed by people who can write complex, coherent essays above 2000 words with like a day of advanced notice. The "missing data type" essay was just 3000 and took me months. Show me your dark magic please.
These conversations about "way back when" and attempting to compare tech opportunities from one time period to another, always seem to focus on the wild opportunities and forget about the equally insane constraints.
Please don't forget that in the late nineties, CPUs were measured in a few hundred MHz, RAM in tens of MBs, and network speeds were kilobytes per second.
FOSS was a new idea and most software cost hundreds or thousands of dollars. People PAID for compilers! Workstations cost many thousands and servers cost millions. These investments would be literally wiped out every year or two by the leading edge of Moore's law.
In the late nineties, I worked for a company that charged $3 million to a big brand to build a website with user forums. It took a year and more than 30 people to launch. We had to buy ~$1m worth of server hardware and fasten it into racks.
Every time constraints change, new opportunities emerge that either weren't possible or just didn't make sense before.
If you're looking for opportunities, keep an eye on observable, recent changes, especially those that haven't yet been broadly distributed.
Look for changes in constraints that are limiting large amounts of valuable activity.
Some random examples in the last 5-10 years (IMO) are:
- LLMs, obviously
- Shared, cloud infrastructure made it trivially inexpensive to launch new business ideas
- High speed computation has fallen in price so much that most applications don't even need cloud infrastructure anymore (IMO)
- The amount of trust many people have in big tech companies is at a nadir, this could be a great time to carve out a beach head in anything they do. Being perceived as trustworthy and NOT THEM could be enough to build a great business.
- Many people seem to feel that social media is net bad for us, figure out how to solve the same problem set (feeling engaged and connected?), with fewer negative trade offs.
- The electronics hobby market has exploded and made new hardware ideas possible, even if budget is limited.
There are a bunch of these kinds of observations that you're uniquely positioned to observe and dig deep on.
Most of them aren't viable, but the fun is in probing around at the edges of what's possible and learning. That's the thing that brings us closer to whatever will work.
Maybe it's just me, but I've noticed that when I'm waking up frustrated and unhappy most days, it has almost always been because I'm ignoring my conscience, which has been trying to tell me that I'm not pointed in the right direction.
he wrote a very interesting c extension language at autodesk called atlast, wrote a diet guide that's made the front page of hacker news countless times, was doing things with neural networks on the commodore 64, had libraries to help make c safe, put some very cool recipes on his website, and also founded some cad company i guess
copycat but not clone of a just-short-of-modular synthesizer before that was a thing (the project he was clean-room reimplementing was by harry pyle, designer of the intel 8008 who he knew personally): https://fourmilab.ch/webtools/MindGrenade/
insanely massive collection of multi-language benchmark results (including an implementation of a raytracer in a range of languages so large it includes algol-60, pl/i & raku)
I've never heard of this until today, but it is more or less the strategy that I've been following for a few years now. I can say it works pretty much as advertised here. I've done some major refactors by splitting into atomic changes like this, and although it takes a lot of time and effort, the end result is comparable to the work put in.
What I mean is that a big refactor might take 9 months to do atomically whereas it could have been done in 3 months as a massive single PR. However I guarantee you we'd have spent 6+ months cleaning up unexpected issues that show up after merging the giant PR. In the atomic commit approach what you get at the end of the 9 months is a solidly engineered product without those issues.
I will say that on a team, issues that you encounter are that it is very hard to keep track of these work-in-progress refactors that are getting in piece by piece, and it can cause rebase hell for other committers if there is a lot of refactoring involved. Splitting one giant refactor into 3 smaller refactors is better for the product, but represents 3x the work for other team members who have to rebase all their changes every time something is merged.
Hello, designer of Ur/Web here. It's great to see it popping up again on HN! Apologies for the diminishment of visible activity in the last few years. I've been distracted by working on our Ur/Web-based startup Nectry (https://nectry.com/) in parallel to my professor job.
We hope to be able to hire more engineers to do Ur/Web development at Nectry in the foreseeable future, so do let me know if you think you might be a good match.
That's exactly how I teach dynamic programming. First solve it recursively, then add memoization (I call that top-down).
Then notice that recursion and memoization has some overhead, and construct the table from bottom-up, remove the recursive call, and voila, dynamic programming.
OK, this is just completely unreasonable of you. HTML is a natural hypermedia in that it has native hypermedia controls. JSON & XML are not natural hypermedia because they do not, however hypermedia controls can be added on top of them, as in the case of HXML/hyperview, which, again I include in my book on hypermedia systems.
There are many other hypermedias, such as Siren, which uses JSON as a base, and I have never claimed otherwise. Mark Amundsen, perhaps the worlds expert on hypermedia, wrote the forward to my book, Hypermedia Systems, and found nothing objectionable and much worthwhile in it.
I hate to be rude but you didn't understand, or refused to acknowledge, the basic meaning and usage of the term 'hypermedia control' until I cited a W3C document using it. While I certainly understand people can dislike the conceptual basis of htmx, its admittedly idiosyncratic implementation or the way we talk about it, at this point I have tried to engage you multiple times in good faith here and have been rewarded with baseless accusations of things I haven't said and don't believe.
At this point, to be an honest person, you need to apologize for misrepresenting what I am saying multiple times to other people. It is dishonest and it makes you a liar, over something as dumb as a technical disagreement.
> Was there a major theoretical development in the solver field that allowed this to happen ?
A few major theoretical developments did happen, although the really big ones are 25+ years ago (see Figure 4 in the OP): 5x in 1994 with the incorporation of the dual simplex method, 10x in 1998, mostly because of cutting planes, Gomory cuts specifically.
> Or is it a bunch of tiny tuning heuristics ?
Also yes. Bob Bixby, co-founder of CPLEX and Gurobi, describes mixed-integer programming as "a bag of tricks". And of course, there is a whole spectrum between pure theory and heuristic trickery, it's not black-and-white.
> If so, how are those even possible given that the solver is supposed to be a generic tool applicable to a large class of problem ?
> Are there problems whose structure fit a recognizable pattern where optimizations are possible ?
Yes, plenty! Commercial solver developers have a business to run. Clients got problems, they need to solve them, regardless of the algorithm. The canonical example of problem-structure-detection is knapsack problems. CPLEX and Gurobi both detect when their input is a knapsack, and they then run a completely different algorithm to solve it.
At a smaller scale (but larger impact overall), there are a wide range of "presolve" techniques that each detect some microstructures in problems and simplify them [1]. Most of these techniques affect <20% of problems, but together they are extremely powerful.
Another example of half-theoretical half-tricky technique that has a great impact on a few instances by detecting structure: symmetry detection. The theory behind it is serious stuff. Implementing the techniques requires serious (and unpublished) engineering efforts. Most problem instances aren't affected at all. But when it works, you can expect a 10x speedup.
When I started two years ago, I basically had to choose a language and a package/framework, so I'd like to share my experience about it.
First, about the package, there are two major ones for 2D games, which are Ebiten(gine) and Pixel.
I initially chose Pixel, because it seemed more documented at the time and had a bigger community. Overall, it was good as long as the game was simple, but unfortunately I regretted this choice later.
Don't get me wrong, I am not here to discredit the author, because there is a phenomenal amount of open source work that was done, and I am thankful for it.
But as my game grew more complex, the engine started to cause big architectural challenges, memory leaks (it does not correctly garbage collect OpenGL's objects for example), and it's multi-threading model is flawed. I also discovered that it was in fact not maintained. Overall it lacks too much maturity for a serious game.
In the end, I decided to ditch it, because it was easier, and my proposals for help on GitHub never got any answer. It took some effort, but having my own engine based on OpenGL was the best choice in this case.
When it comes to Ebiten, I have only used the sound library called Oto, but overall my opinion of the whole package now is that it is well maintained and architected.
Now when it comes to choosing a language, I hesitated between:
- Go, which I had a lot of experience with and seemed good enough, even if the typing system is a bit too weak
- Rust, which I was (and still is) interested in, but had no experience, and unsure if it was worth it in terms of productivity (because of the additional time spent on worrying code, and the compilation time itself). Also, it didn't have game engines which seemed as mature as in Go at this time (I wish I had known that I would end-up with my own engine).
- C++, which is the industry standard, but on which I have very limited experience. Also, I just didn't want to bother with the header files and the compilation stack.
I chose Go, and too be honest it is a choice that I now regret, but also I have to live with it because I cannot afford a rewrite at this point.
Go certainly had a lot of productivity benefits, but also major flaws when it comes to game development:
- The lack of packages related to data structures. In games, we need a lot of specific sets, trees, queues, tries and maps, but those are almost non existent in Go. The best one is "gods", which still does not support generics. Overall it is always difficult to find quality and experienced packages, even for broadly used and fundamental algorithms.
- The fact that dependency cycles are handled on a package basis, and not per-class or per-file. This is a major source of headaches for video games, because contrarily to web servers, there are not a limited number of predefined layers: everything needs to depend on almost everything else, and Go makes it even worse in this case.
- The lack of data structures is made even worse by the native map, because it forcefully randomizes the iteration order, which causes a lot of bugs and makes it unusable 90% of the time. The authors wanted to prevent people from using it as if the key order was guaranteed, but their fix also broke the natural consistency of iteration order. This means for example that the drawing order of objects is different every frame, even when the data didn't change at all.
- When it comes to modding, there are not much options, because Go does not have a very easy or natural compatibility with FFI (cgo is awkward to use, and not really usable with dynamic linking), the native plugin package is experimental (and not even working in windows), and the support for WebAssembly is still quite poor (but getting better).
I'm making a game in Go, and it runs perfectly smooth on my 240hz monitor (and should run perfectly smooth on 1000hz monitors once we get there).
The garbage collector has not been an issue, because I use memory the same way as if I were doing it in the C language: Allocate all memory used by the game at program launch. I find this scheme easier to work with in Go than in C, because Go's slice type is a natural fit for partitioning these launch-allocated blocks of memory (as they get used and reused during the game).
The garbage collector doesn't get called if you don't allocate. And for the most part¹, it's easy to reason about where Go allocates (if you know C). So my rule is: Always know where your memory is; never allocate.
This may sound like it's going against the grain of Go, but I find Go to be handily amenable to this style. I had my concerns going in, but it just hasn't been a problem. My game runs silky smooth.
At this point, my only concern about using Go to make a game is that bringing the game to consoles does not have a well-trodden path.
¹ So far, the only thing that has surprised me is that assigning a value type (like an int) to an interface allocates! (So another rule is to only assign pointers to interfaces [pointers to memory allocated at launch].) And while this was a surprise to me, I was pleased with how quick the debugging went: I noticed frames were dropping, so I ran go's pprof tool, which led me directly to the (freshly coded) interface assignment that was stealth allocating. (Also, this allocation becomes less surprising the more I think about how interfaces are implemented.)
Yes, the mental fortitude is much overblown aspect.
Yes, there is this thing as mental fortitude and you can even train it to persist to run hard, especially when you "kick" at the end of a race.
But this only helps to a very small degree, the most of the result is due to more mundane aspects like your experience with running, your running efficiency, your aerobic capacity, your additional fat you are carrying with you, whether you have been running regularly recently, etc.
I am regular runner (10 miles every morning). If I have an injury and are out of running for two months, I will have very hard time at the same speed and duration as before the injury and no mental fortitude will help me get the same result come race time. My mental fortitude counts for less than my last two months of running history.
Even just a tiny bit of experience with running can dramatically improve your results. And that is not necessarily by improving your fitness. I remember when I started running I would get large increases in results every single day. I was very proud of myself until somebody told me this is normal and has nothing to do with my fitness -- my brain is getting rewired to coordinate my movements better and this helps running efficiency. Another important aspect of running you learn within first days is how it feels to be running at a sustainable pace. If you don't have any experience you are very likely to run too fast, accumulate lactate and be unable to finish the distance or have to slow down considerably. Because you don't have any idea how hard is appropriate for the challenge and your ability.
And one more argument against mental fortitude. I think it has very little value the better you are at running. The better you are at running the closer you run to your physical limits and once you understand it, the brain alone cannot make you run faster if your body simply is not able to produce required output.
If you observer Eliud Kipchoge, for example, beating yet another marathon record, you will notice he is feeling good and smiling throughout the race and at the finish line. Yes, there is probably some pain and discomfort but really, he is just executing a meticulously prepared plan. The work has already been done and he can't run faster than his ability and he knows is ability to perfection. He knows if he runs even a tiny bit faster than he should, it will actually put him above his ability and produce a slower result. There is maybe last couple miles where he is judging his reservers and decides whether to run a tiny bit faster or stay at his plan and that's about it.
I had to use mental fortitude to finish my first marathon (when my body cut the power at 20 miles), but that's just because I was undertrained and unprepared. Now when I run I know perfectly the pace I need to keep to finish the race and there isn't temptation to "push it" because I know if I do this it will produce a worse result.
(yes, I know running long distance like 5k-42.2k is very different from sprinting. I am just an amateur who runs a lot who is also a nerd and want to understand a lot about everything I do.)
> you will be exploited for amplification attacks!
Is that the gripe? Am I missing something?
Your bellybutton is not subject to these amplification attacks, it is strangers on the internet and this is an altruistic appeal: you will be exploited for amplification attacks directed at others.
What is the root cause of this vulnerability? The root cause is that anonymous[0] packets on the internet contain a destination address the packet is to be delivered to, and a source address to which responses should be directed. That might be a different construction than the average programmer internalizes. In the absence of countermeasures either address can be rewritten and this is commonly done in firewalls with e.g. NAT.
The attack is to specify a source address "on behalf of" some other party, causing replies to be delivered there. Are there other ways to validate or render packets as not anonymous? Yes, to varying degrees. One is to establish a dialogue with the source address before sending payloads; in fact that's pretty much the good which abnegates perfection.
There has been a BCP for many years that ASNs should validate that source addresses are valid.[1] Amplification is possible both in velocity and volume. Volume is when a reply packet is larger than the source packet. Velocity occurs when mulitple replies are sent. Even when no amplification occurs, attacks are mounted with e.g. ICMP. UDP amplification typically occurs when the reply packet is larger than the initiating packet. TCP amplification occurs when a SYN can elicit multiple SYN/ACKS.[3] (Both TCP and UDP are capable of eliciting ICMP backscatter.)
The simple way to avoid amplification with UDP is not to send a reply. This is done a lot. For instance streaming data over UDP with a TCP pipe for control information. A bonus for this approach is that UDP is conducive to multicast[4], for which specific address ranges are reserved.
Changing tuning parameters is an effective way to mitigate the TCP SYN/ACK problem, barring the appearance of flying monkeys.[5]
DNS may be illustrative in a number of ways. First off clients aggressively retry queries due to "happy eyeballs", so the servers constantly face an identify friend or foe operating environment. DNS only ever sends one response to a particular query. A common server (if not protocol) extension is the implementation of response rate limiting, which drops a certain portion of UDP replies and returns the rest with TC=1.[6]
Congratulations on reading this far. Now go get yourself a job as a cybersecurity analyst.[7]
[0] If you trust the source in some other fashion and mark or validate packets in some way based on that, they're not anonymous.
There's an even simpler solution I've used ever since I became aware of the first amplification attacks during the 00s: either implement your own simple 3wh, or if you really want a simple request/reply protocol with no state associated, pad the request so it matches the reply in size. Sometimes your reply is large though and needs a couple dozen packets so then you're usually forced to use the first solution.
That has parallels with Escher’s ‘paradoxical’ image of two hands recursively drawing each-other: it’s not paradoxical once you realise a third hand drew them sequentially in order for them to appear paradoxical.
For source code, I prefer the Software Heritage archive over the Internet Archive, because it archives the git history, instead of the HTML UI. This particular repo was saved there[0] and was most recently visited 24 Oct 2022, which has two more terms updates.
> It did make me very hesistant and perfectionist in future projects and took out a certain amount of faith that 'it will work out'.
But that's because you still cared about others. And maybe didn't read enough startup/entrepreneur books or know enough people who did that. Once you know what crap companies willingly release which go on to be successful, it will make you far less perfectionist.
I had the luck of having a few family members in computing in the 70s when I was born, so I heard early on about Allen debugging/fixing the basic interpreter he wrote on a PDP for a processor he didn't have access to at the time on the plane for one of the most important meetings of Microsoft at the time. The absolute and total garbage Oracle released for many years when they started. etc. And those are old stories, but this happens all the time; a lot of products that are launched by big or small companies are so full of bugs that I wonder sometimes if anyone bothered to try them.
I simply stopped caring about perfection as it drove me insane and no-one cares anyway. Even, in line with this article, even if I make something ONLY for myself, I rather just finish it and then change it over time then try to get it in a 'perfect state' from the start. It doesn't make me happy as perfection is not really something that's easy to define; especially in software, it is so incredibly hard to get to anything better than 'ok', that striving for perfection (Dijkstra like) will drive you completely bonkers and nothing ever gets released.
First thing I do, is a 5K brisk walk. Gets the exercise quota satisfied. I tried running, but kept injuring myself, so walking, it is. I can do that, seven days a week. Takes a bit less than an hour (I walk about 10min/Km).
I usually "triage" my day, on these walks. I mentally sort through the tasks ahead, and scope out my plans. Sometimes, it's straightforward, other times, I have to develop a plan.
When I get home, I hit the shower, and generally start coding, right away. I code throughout the day, taking breaks as I need/want. I have no issue, stopping work for a couple of hours, and watching some TV or even taking a short nap.
I'll code at any time. I don't have a "set" workday.
I have found that I am most productive in the morning, and most creative at night, but I sometimes find myself losing focus, towards the end of the day. When that happens, I generally wrap things up, as I do more damage than good. I'll often fix a bug in my head during my walk, that was confounding me the night before.
I believe in F^3 - Finite, Focused, and Finished.
- Finite: I know what "done" looks like. Tasks have a discrete beginning and end, though the schedule may be fuzzy.
- Focused: No distractions. I stay on beam, and devote full attention to the task at hand.
- Finished: I don't stop, until I have reached a point that I can wrap things. I usually punctuate this with a final tagged commit.
It's also important to scope things reasonably. Stretch, but not too much.
I think it's easy to see that psychedelics, while valuable, aren't exactly a direct key to world peace. Why? Because we won't have peace in the world until we have peace in people. And psychedelics, whatever they do, and I think they do something very useful, do not make people peaceful. Even if a person has many psychedelic experiences over a long period of time, they will most likely still be afflicted to some degree by greed, envy, anger, resentment, fear, etc.
When we say we want peace in the world, we really mean we want peace in the human world. And the human world is made of humans. But each person wants peace to be achieved by all other humans making themselves peaceful -- very few people have any practice to make themselves more peaceful.
> That aside, operator overloading is fits the specific "user glue" you are focusing on
I thought so as well, for a long time, but it turns out not to be the case, because you can't properly do "connection" via call/return. You can do it, but not "properly".
The Problem: Architectural Mismatch
------
The crux is that all mainstream and virtually all non-mainstream languages aren’t really general purpose, though we obviously think of them and use them that way. They are DSLs for the domain of algorithms, for computing answers based on some inputs. Obviously with many variations, but the essence is the same. a := f(x). Or a := x.f(). Or a: x f. And the algorithmic mechanism we have for computing answers is also our primary linguistic mechanism for structuring our programs: the procedure/function/method. (Objects/classes/prototypes/modules are secondary mechanisms).
(see also: the ALGOrithmic Language, predecessor to most of what we have, and allegedly an improvement on much of that).
As long as our problems were also primarily algorithmic, this was perfectly fine. But they no longer are, they are shifting more and more away from being algorithmic.
For example, if you look at how we program user interfaces today, there hasn’t really been much progress since the early 80s or 90s. Arguably things have actually taken huge leaps backwards in many areas. I found this very puzzling, and the common explanation of “kids these days” seemed a bit too facile. The problem became a bit clearer when I discovered Stéphane Chatty’s wonderful, and slightly provocatively named paper "Programs = Data + Algorithms + Architecture: consequences for interactive software engineering".
He explains very well, and compellingly, how our programming languages are architecturally mismatched for writing GUIs.
Once you see it, it becomes really hard to un-see, and this problem of algorithmic programming languages, or in the architectural vernacular I prefer, call/return programming languages pops up everywhere.
Or how Guy Steele put it:
"Another weakness of procedural and functional programming is that their viewpoint assumes a process by which "inputs" are transformed into "outputs"; there is equal concern for correctness and for termination (and proofs thereof). But as we have connected millions of computers to form the Internet and the World Wide Web, as we have caused large independent sets of state to interact–I am speaking of databases, automated sensors, mobile devices, and (most of all) people–in this highly interactive, distributed setting, the procedural and functional models have failed, another reason why objects have become the dominant model. Ongoing behavior, not completion, is now of primary interest."
---
I also thought plain old metaprogramming facilities would do the trick, but in the end those are also insufficient. (For example, my HOM was entirely done via metaprogramming).
Architecture Oriented Programming is a generalisation of call/return programming, not an extension.
I’m a big fan of the Donald Reinertsen approach: measure queue length.
Simply track the time to complete each task in the team queue on average, then multiply that by the number of tasks remaining in the queue.
Each team will habitually slice things into sizes they feel are appropriate. Rather than investing time to try and fail at accurately estimating each one, simply update your average every time a task is complete.
The bonus with this approach is that the sheer number of tasks in the queue will give you a leading indicator, rather than trailing indicators like velocity or cycle time.
I did a part of the incremental parsing in Menhir and the whole recovery aspect. I can try to explain a bit.
My goal was Merlin (https://github.com/ocaml/merlin/), not research so there is no paper covering the work I am about to describe. Also as of today only incrementality and error message generation are part of upstream version of Menhir, but the rest should come soon.
# Incrementality, part I
The notion of incrementality that comes builtin with Menhir is slightly weaker than what you are looking for.
With Menhir, the parser state is reified and control of the parsing is given back to the user. The important point here is the departure from a Bison-like interface. The user of the parser is handled a (pure) abstract object that represents the state of the parsing.
In regular parsing, this means we can store a snapshot of the parsing for each token, and resume from the first token that has changed (effectively sharing the prefix).
But on the side, we can also run arbitrary analysis on a parser (for error message generation, recovery, syntactic completion, or more incrementality...).
# Incrementality, part II
Sharing prefix was good enough for our use case (parsing is not a bottleneck in the pipeline). But it turns out that a trivial extension to the parser can also solve your case.
Using the token stream and the LR automaton, you can structure the tokens as a tree:
- starts with a root node annotated by the state of the automaton
- store each token consumed as a leaf in that tree
- whenever you push on the automaton's stack, enter a sub-level of the tree (annotated by the new state), whenever you pop, return to the parent node
This gives you the knowledge that "when the parser is in state $x and $y is a prefix of the token stream, it is valid to directly reduce the subtree $z".
In a later parse, whenever you identify a known (state number, prefix) pair, you can short-circuit the parser and directly reuse the subtree of the previous parse.
If you were to write the parser by hand, this is simply memoization done on the parsing function (which is defunctionalized to a state number by the parser generator) and the prefix of token stream that is consumed by a call.
In your handwritten parser, reusing the objects from the previous parsetree amounts to memoizing a single run (and forgetting older parses).
Here you are free to choose the strategy: you can memoize every run since the beginning, devise some caching policy, etc (think about a user (un)commenting blocks of code, or switching preprocessor definitions: you can get sharing of all known subtrees, if this is of any use :)).
So with part I and II, you get sharing of subtrees for free. Indeed, absolutely no work from the grammar writer has been required so far: this can all be derived by the parser generator (with the correctness guarantees that you can expect from it, as opposed to handwritten code).
A last kind of sharing you might want is sharing the spine of the tree by mutating older objects. It is surely possible but tricky and I haven't investigated that at all.
I might be biased but contrary to popular opinions I think that LR grammars are well suited to error message generation.
The prefix propery guarantees that the token pointed out by the parser is relevant to the error. The property means that there exist valid parses beginning with the prefix before this token. This is a property that doesn't hold for most backtracking parsers afaik, e.g PEG.
Knowledge of the automaton and grammar at compile time allow a precise work on error messages and separation of concerns: the tooling ensures exhaustivity of error coverage, assists in migration of error messages when the grammar is refactored, or can give an explicit description of the information available around a given error.
This is not completely free however, sometimes the grammar needs to be reengineered to carry the relevant information. But you would have to do that anyway with a handwritten parser and here the parser generator can help you....
If you have such a parser generator of course :). Menhir is the most advanced solution I know for that, and the UX is not very polished (still better than Bison). It is however a very officient workflow once you are used to it.
So LR is not the problem, but existing tools do a rather poor job at solving that kind of (real) problems.
# Recovery
In Merlin's, recovery is split in two parts.
The first is completion of the parsetree: for any prefix of a parse, it is possible to fill holes in the tree (and thus get a complete AST).
This is done by a mix of static analysis on the automaton and user guidance: for major constructions of the AST, the grammar writer provide "dummy" constructors (the node to use for erroneous expressions, incomplete statement, etc).
The parser generator then checks that is has enough information to recover from any situation, or point out the cases it cannot handle.
It is not 100% free for the grammar writer, but for a grammar such as OCaml one it took less than an hour of initial work (I remember only a few minutes, but I wasn't measuring properly :)). It is also a very intuitive step as it follows the shape of the AST quite closely.
The second part is resuming the parse after an error (we don't fill the whole AST every time there is an error; before filling holes we try to look at later tokens to resume the parse).
There is no big magic for that part yet, but the heuristic of indentation works quite well: constructions that have the same indentation in source code are likely to appear at the same depth in the AST, and this is used to decide when to resume consuming tokens rather than filling the AST.
I have a few lead for more disciplined approaches, but the heuristic works so well that it isn't an urgency.
# Conclusion
I would say that if I had to write a parser for a language for which a reasonable LR grammar exists, I will use a (good) parser generator without hesitation.
For the grammars of most programming languages, LR is the sweet spot between parsing expressivity and amenability to static analysis.
When the performance overhead of having a GC is less significant than the cognitive overhead of dealing with manual memory management (or the Rust borrow checker), Go was quite successful: Command line tools and network programs.
Around the time Go was released, it was certainly touted by its creators as a "systems programming language"[1] and a "replacement for C++"[2], but re-evaluating the Go team's claims, I think they didn't quite mean in the way most of us interpreted them.
1. The Go Team members were using "systems programming language" in a very wide sense, that include everything that is not scripting or web. I hate this defintion with passion, since it relies on nothing but pure elitism ("Systems language are languages that REAL programmers uses, unlike those "Scripting Languages"). Ironically, this usage seems to originate from John Ousterhout[3], who is himself famous for designing a scripting language (Tcl).
Ousterhout's definition of "system programming language" is: Designed to write applications from scratch (not just "glue code"), performant, strongly typed, designed for building data structures and algorithms from scratch, often provide higher-level facilities such as objects and threads.
Ousterhout's definition was outdated even back in 2009, when Go was released, let alone today. Some dynamic languages (such as Python with type hints or TypeScript) are more strongly typed than C or even Java (with its type erasure). Typing is optional, but so it is in Java (Object), and C (void*, casting). When we talk about the archetypical "strongly typed" language today we would refer to Haskell or Scala rather than C. Scripting languages like Python and JavaScript were already commonly used "for writing applications from scratch" back in 2009, and far from being ill-adapted for writing data structures and algorithms from scratch, Python became the most common language that universities are using for teaching data structures and algorithms! The most popular dynamic languages nowadays (Ruby, Python, JavaScript) all have objects, and 2 out of 3 (Python and Ruby) have threads (although GIL makes using threads problematic in the mainstream runtimes). The only real differentiator that remains is raw performance.
The widely accepted definition of a "systems language" today is "a language that can be used to systems software". Systems software are either operating systems or OS-adjacent software such as device drivers, debuggers, hypervisors or even complex beasts like a web browser. The closest software that Go can claim in this category is Docker, but Docker itself is just a complex wrapper around Linux kernel features such as namespaces and cgroups. The actual containerization is done by these features which are implemented in C.
During the first years of Go, the Go language team was confronted on golang-nuts by people who wanted to use go for writing systems software and they usually evaded directly answering these questions. When pressed, they would admit that Go is not ready for writing OS kernels, at least not now[4][5][6], but GC could be disabled if you want to[7] (of course, there isn't any way to free memory then, so it's kinda moot). Eventually, the team came to a conclusion that disabling GC is not meant for production use[8][9], but that was not apparent in the early days.
Eventually the references for "systems language" disappeared from Go's official homepage and one team member (Andrew Gerrand) even admitted this branding was a mistake[10].
In hindsight, I think the main "systems programming task" that Rob Pike and other members at the Go team envisioned was the main task that Google needed: writing highly concurrent server code.
2. The Go Team members sometimes mentioned replacing C and C++, but only in the context of specific pain points that made "programming in the large" cumbersome with C++: build speed, dependency management and different programmers using different subsets. I couldn't find any claim that go was meant as a general replacement for C and C++ anywhere from the Go Team, but the media and the wider programming community generally took Go as a replacement language for C and C++.
When you read through the lines, it becomes clear that the C++ replacement angle is more about Google than it is about Go. It seems that in 2009, Google was using C++ as the primary language for writing web servers. For the rest of the industry, Java was (and perhaps still is) the most popular language for this task, with some companies opting for dynamic languages like Python, PHP and Ruby where performance allowed.
Go was a great fit for high-concurrency servers, especially back in 2009. Dynamic languages were slower and lacked native support for concurrency (if you put aside Lua, which never got popular for server programming for other reasons). Some of these languages had threads, but these were unworkable due to GIL. The closest thing was frameworks Twisted, but they were fully asynchronous and quite hard to use.
Popular static languages like Java and C# were also inconvenient, but in a different way. Both of these languages were fully capable of writing high-performance servers, but they were not properly tuned for this use case by default. The common frameworks of the day (Spring, Java EE and ASP.net) introduced copious amounts of overhead, and the GC was optimized for high throughput, but it had very bad tail latency (GC pauses) and generally required large heap sizes to be efficient. Dependency management, build and deployment was also an afterthought. Java had Maven and Ivy and .Net had NuGet (in 2010) and MSBuild, but these where quite cumbersome to use. Deployment was quite messy, with different packaging methods (multiple JAR files with classpath, WAR files, EAR files) and making sure the runtime on the server is compatible with your application. Most enthusiasts and many startups just gave up on Java entirely.
The mass migration of dynamic language programmers to Go was surprising for the Go team, but in hindsight it's pretty obvious. They were concerned about performance, but didn't feel like they had a choice: Java was just too complex and Enterprisey for them, and eeking out performance out of Java was not an task easy either. Go, on the other hand, had the simplest deployment model (a single binary), no need for fine tuning and it had a lot of built-in tooling from day one ("gofmt", "godoc", "gotest", cross compilation) and other important tools ("govet", "goprof" and "goinstall" which was later broken into "go get" and "go install") were added within one year of its initial release.
The Go team did expect server programs to be the main use for Go and this is what they were targeting at Google. They just missed that the bulk of new servers outside of Google were being written in dynamic languages or Java.
The other "surprising use" of Go was for writing command-line utilities. I'm not sure if the original Go team were thinking about that, but it is also quite obvious in hindsight. Go was just so much easier to distribute than any alternative available at the time. Scripting languages like Python, Ruby or Perl had great libraries for writing CLI programs, but distributing your program along with its dependencies and making sure the runtime and dependencies match what you needed was practically impossible without essentially packaging your app for every single OS and distro out there or relying on the user to be a to install the correct version of Python or Ruby and then use gem or pip to install your package. Java and .NET had slow start times due to their VM, so they were horrible candidates even if you'd solve the dependency issues. So the best solution was usually C or C++ with either the "./configure && ./make install" pattern or making a static binary - both solutions were quite horrible. Go was a winner again: it produced fully static binaries by default and had easy-to-use cross compilation out of the box. Even creating a native package for Linux distros was a lot easier, so all you add to do is package a static binary.
[1]: https://opensource.googleblog.com/2009/11/hey-ho-lets-go.htm...
[2]: https://web.archive.org/web/20091114043422/http://www.golang...
[3]: https://users.ece.utexas.edu/~adnan/top/ousterhout-scripting...
[4]: https://groups.google.com/g/golang-nuts/c/6vvOzYyDkWQ/m/3T1D...
[5]: https://groups.google.com/g/golang-nuts/c/BO1vBge4L-o/m/lU1_...
[6]: https://groups.google.com/g/golang-nuts/c/UgbTmOXZ_yw/m/NH0j...
[7]: https://groups.google.com/g/golang-nuts/c/UgbTmOXZ_yw/m/M9r1...
[8]: https://groups.google.com/g/golang-nuts/c/qKB9h_pS1p8/m/1NlO...
[9]: https://github.com/golang/go/issues/13761#issuecomment-16772...
[10]: https://go.dev/talks/2011/Real_World_Go.pdf (Slide #25)