I think this is something that happens to any field.
I trained as a 2D animator. When you start, it's great - there's just you and the drawings. You can make anything happen. But once you start to want to make something bigger than a couple of seconds, things get complicated; you have to plan a script, wrangle multiple scenes, start involving other people to break up the process in various ways - inbetweeners fill in frames between what the lead animator draws, cleanup artists neaten things up, colorists, background painters, 3D people for stuff that's a pain to do by hand... plus all the non-drawing support people. Watch the credits at the end of any animated feature. All of them. Every single one of those people put in a lot of hours to make that film happen; not all of them did work that shows up directly on the screen. But all of those people had an important part to play, and all of them had to have ways to communicate with each other.
I burnt out on that because I didn't want to be a cog in a machine designed to tell stories to the largest possible market. Now I draw comics. And guess what? I've got to deal with learning how to put together a website, put together a book for print, promote my work, go to conventions and sell stuff, etc. It is just barely doable by one person; I've been spending three years doing a ~400p graphic novel by myself, and I've got about a half a year left before it's done. I have template InDesign documents, scripts to help fill those templates, friends willing to try to make sense of the in-progress work and tell me when things don't make sense, sprawling directory structures full of source files and web/print res files, sketchbooks full of planning, etc etc etc. It helps that I was already able to wrangle a website; I'm slowly turning my custom templates into something anyone who wants to present a comic online the same way I am can use. If I wanted to start telling a lot more stories a lot more quickly, I could start parallelizing: separate the job into writer/artist/colors/lettering, find an existing publisher willing to handle packaging/printing/distributing to comic stores, bookstores, and e-stores, as well as shop it around to Hollywood.
I can knock out a nice standalone drawing in a few hours. I still do that every now and then when I need a break. But my aspirations are higher; I want to create a world for the reader to inhabit for a few hours. And that takes a lot more work.
Every kind of complex project has its own kind of scaffolding that users of the final project will never see.
Good process makes your work of being "a cog in the machine" much more easier (and abstracts the rest for you - and of course, your work will be abstracted away for other people)
This is true. The difference, however, is that software development is often considered to be largely an art centered around managing complexity.
While we're generally OK at doing this in the scope of one piece of software, we suck at doing it in workflow terms, particularly where other humans, inter-organizational dependencies, money and existing systems are involved. What this article is really talking about is shifting the complexity of the varying development and deployment toolchains and processes adopted by each different project and organization away from the developer.
The author cites a fairly simple example from the commercial programming world. The complexities he refers to are version control systems, editors/IDEs, build tools, language selection, database/persistence layer selection, test deployment, bug tracking, release processes.
Let's assume learning basic git, how to use web-based issue trackers and a text editor are reasonably considered non-negotiable, per-programmer overhead. The author still has a fair point that IDEs can be a plague in some sectors (mobile and Java in particular).
He is also correct that build tools are numerous, but what makes them so painful? Doug McIlroy famously stated Keep it simple, make it general, and make it intelligible. Is part of the pain that many of them are language-specific, with zero or poor integration with external (eg. system-wide) software package databases / dependency trees? It seems that, today, every operating system distribution and every programming language running upon them generally has (at least) one half-baked, error-prone, cross-platform package management solution with its own versioning scheme, configuration overhead, network assumptions, poorly implemented caching system and probably a broken use of cryptographic primitives. They're not general.
Database/persistence layer selection is an architectural question that is easy to either delay or perhaps - if extensive performance tuning is certainly expected, or deployment environments will certainly be heterogeneous in their persistence layers - sidestep through a layer of abstraction (CRUD) to facilitate provider abstraction. These paths are considered good practice - Ken Thompson teaches us to Spell creat with an 'e' (ie. premature optimization is a fallacy), Eric S. Raymond restates Prototype before polishing. Get it working before you optimize it. and RFC3439 summarizes most eloquently Optimization considered harmful: In particular, optimization introduces complexity, and as well as introducing tighter coupling between components and layers.
The final points of pain (test deployment, release processes) are essentially process design and infrastructure interaction and management issues. While the author mentions the additional complexity of the cloud, in truth this area has become a point of pain so intense that many projects now write their code and internal test/release processes specifically targeting one infrastructure provider (EC2, docker, etc.). That's not very general either, now is it?
So where is there clear room for improvement? The build tool issues are mostly a convention versus configuration question, which is philosophical and not going away. The process/infrastructure issues do seem to be the main candidates for change, and this does appear to be the focus of the author's call for approaches.
Tesler's Law of Conservation of Complexity: Every application has an inherent amount of irreducible complexity. The only question is who will have to deal with it - the user, the application developer, or the platform developer? - Larry Tesler (ca. 1984)
One could take the same view of the development process itself. Well, then... who are we pushing the complexity to? Is it...
(1) a single cloud provider of choice (EC2)?
(2) a platform-specific virtualization wrapper of choice (docker)?
(3) a person or team within the organization who just delegates truth from above (BOFH-style)?
(4) Some lightweight, open source workflow solution that enables us to step beyond the provider/technology lock-in of (1) or (2) and remains painless and configurable enough for mass adoption?
I believe that (1) and (2) do not meet the complexity requirements of larger scale users, and that (3) is not scalable or efficient. I think (3) will exist, but focus more on policy, whereas (4) style solutions will appear in the same vein as the literate, simplistically ops-oriented command line interfaces championed most recently by heroku, vagrant and docker. Regarding (4), if we get it right, we don't even need to notice the implementation. As Minsky said: We're more aware of simple processes that don't work well than of complex ones that work flawlessly..
I trained as a 2D animator. When you start, it's great - there's just you and the drawings. You can make anything happen. But once you start to want to make something bigger than a couple of seconds, things get complicated; you have to plan a script, wrangle multiple scenes, start involving other people to break up the process in various ways - inbetweeners fill in frames between what the lead animator draws, cleanup artists neaten things up, colorists, background painters, 3D people for stuff that's a pain to do by hand... plus all the non-drawing support people. Watch the credits at the end of any animated feature. All of them. Every single one of those people put in a lot of hours to make that film happen; not all of them did work that shows up directly on the screen. But all of those people had an important part to play, and all of them had to have ways to communicate with each other.
I burnt out on that because I didn't want to be a cog in a machine designed to tell stories to the largest possible market. Now I draw comics. And guess what? I've got to deal with learning how to put together a website, put together a book for print, promote my work, go to conventions and sell stuff, etc. It is just barely doable by one person; I've been spending three years doing a ~400p graphic novel by myself, and I've got about a half a year left before it's done. I have template InDesign documents, scripts to help fill those templates, friends willing to try to make sense of the in-progress work and tell me when things don't make sense, sprawling directory structures full of source files and web/print res files, sketchbooks full of planning, etc etc etc. It helps that I was already able to wrangle a website; I'm slowly turning my custom templates into something anyone who wants to present a comic online the same way I am can use. If I wanted to start telling a lot more stories a lot more quickly, I could start parallelizing: separate the job into writer/artist/colors/lettering, find an existing publisher willing to handle packaging/printing/distributing to comic stores, bookstores, and e-stores, as well as shop it around to Hollywood.
I can knock out a nice standalone drawing in a few hours. I still do that every now and then when I need a break. But my aspirations are higher; I want to create a world for the reader to inhabit for a few hours. And that takes a lot more work.
Every kind of complex project has its own kind of scaffolding that users of the final project will never see.