I think the "10x" myth persists because we all imagine that we're the programmers who are "10x" better than average.
It's nominally true because many programmers have zero or lower productivity, sometimes due to no fault of their own. If you work on a project for three years, and then it gets canceled, you could say your work had no economic value because it never got in front of customers. If some PHP hack manages to pound something out in a month that brings in $100,000 of business, he looks like a hero, no matter how bad the code is.
In my view, superproductivity is about alignment with your environment. If you work for 12 months a year, but spend 3 of them on side projects that go nowhere, spend 3 months being a sysadmin, and then waste another 5 months on reworking a GUI, there goes about 90% of your productivity.
The "Mythical Man Month" says that about 20% of the time on a software project is spend coding, the rest is requirements work, testing and that kind of stuff. A superprogrammer with supertools might be able to do that in time that approaches zero asymptotically, but unless you can get rid of the other 80%, project cost and time don't get much worse, even in comparison with a coder who is half as productive as average.
The moral? If you want to look like a genius, find some place where (i) requirement work takes little time (and wastes little time downstream with changes) and (ii) testing, deployment and all that is minimal.
On some weeks, I am 10x.. no, scratch that, 50x more productive than myself on the others. And I've seen some people who are capable to sustain that indefinitely.
I think the big productivity improvements are those that in retrospect just seem like: "Yeah, that was the right way to do things."
Example: I had some large csv files to import into a rails app. Instead of writing my own importer and running it with script/runner, I scripted a quick importer ontop of mysqlimport. Mysqlimport ran about 1000x faster than a custom solution.
Well when it concerns yourself it's a bit easier to gauge performance. I'm an embedded systems developer, sometimes able to write 5 device drivers in 5 days, but on others barely able to begin, let alone complete, even the most trivial tasks.
I guess this averages out. Just that some people seem to have incredible mental stamina.
I have seen programmers that are more than ten times faster than some slow programmers. There are debugging tasks where some people struggle for days, while others would find it in one hour.
Sure, which leads one to thinking about it like auto racing.
You don't win a race by "driving fast", you win a race by never driving slowly. It doesn't matter how fast your car is if you spend 30% of your time stacked up at intersections.
Similarly, programmers spent a lot of time dealing with various barriers to productivity. (For instance, bugs you can't fix) If you want to increase your productivity, you need to attack these barriers by every means necessary.
Take any non-trivial debugging task (i.e. something that slipped through the net for several releases in a live system, not a simple bug in fresh code) and set me on it 100 times, and you'll almost definitely see at least 10x variation in how long it takes me, and that's just because of the nature of debugging, sometimes you get lucky and sometimes you don't.
Of course, there's some real variation in the averages that people will settle down to over the course of hundreds of debugging tasks, but an individual's productivity on debugging tasks varies so much that I'd be hard pressed to say much at all about the individuals until I'd seen them do at least a few trials.
It may be the case that the differences in the average times people take do end up being on the order of 10x or more, but it would take a lot of observation to say that for sure (exactly how much depends on what sort of distributions we see when we measure this stuff).
Kudos for pointing out that it's the shape of the distribution that matters, not how far apart its endpoints.
Interestingly there are articles out there which suggest that one distribution (of competence, rather than productivity, but intuitively you'd expect the two to be related) is actually bimodal rather than normal, "the camel has two humps":
It's not that it is unproductive (for me), but rather that it feels like drudge work. Which makes me less motivated to do it, which negatively influences my productivity on other parts of the project. I think we can add documentation to this as well.
Old school people might call it getting stuck in a rut. It's much easier for me to get stuck in a rut doing testing and documentation, compared to doing something new and exciting.
Fair enough, but classifying something as "unproductive" just because you don't enjoy doing it seems a bit... incorrect.
I'm curious as to whether developing automated testing and deployment stuff would count as doing testing and deploying, or as developing exciting new stuff.
Sorry, just to clarify a bit: it's not that I consider it unproductive because I don't like it, but because my personal productivity is much less for projects I am less engaged in. So I'm less productive due to the nature of the work.
And for me, developing the automated stuff is still not particularly interesting. I guess if I were designing cars, my interests (and thus most productive department) would lie with building the car itself (engine, body, drivetrain, etc.) rather than the safety system or QA. Maybe it's just my personality.
It's important to note that (as far as I can tell) the author doesn't actually debunk the claim that good programmers are 10x more productive, only the claim that studies show that programmers are 10x as productive.
Personally, I think the author is unintentionally also pointing out the problem of relying on studies of programmer productivity: it's next to impossible to measure. I seriously doubt that there is no valid research on this because no one has gotten around to it. Too many peoples' businesses rely on understanding programmer productivity inside and out for that to be the case.
Personally, I agree with whoever it was that argued that we need to view this as a soft science like psychology or sociology. We need to focus on working in spite of our imperfect means of measuring productivity rather than dismissing all of our research based on it. Such is the nature of studying the human mind.
I read quite a bit about measurable programmer productivity in the 90's because it was an interest of mine. Basicly the studies were worthless. Professional programmers were too expensive, so CS undergraduates were used. As you can imagine there are 10x differences in programmer experience among undergraduates. When my son was interviewing colleges a decade or so ago, we sat down and estimated that he had written about 40,000 lines of code in high school, mostly professionally and mostly hard stuff, like video codecs, mone was easy stuff like html or php. Some of you have done that. Some of his classmates had done some basic a summer or two before. Well there is at least 10x that he could easily maintain. 10 years after college there are still 10x programmers, but things are much moir even.
The bottom line? Studies using a group of undergraduates on a day-long project are worthless.
The other thing I have seen are efforts to measure professional programmers via time sheets, even had to do it myself, usually in the guise of getting us to produce more reliable estimates. Not more accurate estimates, but more reliable in the sense that Jack is always off by 4x while Jill is consistently off by 1.5x. These efforts usually died because programmers resisted the hassle.
Other measurements have been equally worthless. Measure by lines of code? You get lots of blank line and comments. I have even worked on projects that required 2 pages of comment for 5 lines of code.
The real issue is studding programmers is expensive. In class assignments or contests you can find a huge range of time to completion. But, paying a statistically significant number of programmers to solve the same task is prohibitively expensive let alone a wide range of identical tasks.
PS: I have completed the same task as other programmers in less than 10% of the time. However, assessing overall productivity requires that someone is consistently faster on a wide range of tasks not just that they happen to solve something in 10% or even 1% of the time.
I dunno, introductory level programming classes create an opportunity to observe anywhere from 5 to 200 people solving the same problem.
I've never taught a CS course, but back in my college days I helped a friend with his CS homework by taking advantage of weak file permissions to copy other people's homework. The shocking thing was that, looking at a small sample, most of the programs didn't work correctly. To complete the assignment our way, we had to fix bugs in programs that other people wrote, which is fantastic preparation for a professional programmer ;-)
But using introductory programming courses as an example of writing real-world code would be a huge mistake. Most of those assignments in my day were almost trivially easy for someone that actually understands the material (and almost impossible by those who don't). It would not be a fair assessment of software engineering productivity.
Worse, the homework assignments are generally completed outside of class under widely ranging conditions. (One student finds a quiet spot in a library, another is in a cramped dorm room with a roommate practicing trumpet, and another goes out and parties until dawn and tries to crank out a project three hours before it's due.)
In your attempt to weed out conflating variables, you introduce a similar variable... namely: all programmers working in the same conditions definitely reduces external stimuli which could impact performance, however it introduces a different variable in which the some (or many!) of the programmers are no longer working under their preferred conditions dropping their performance.
How much of your speed in completing a task based on prior experience with similar or identical problems with your experience with the programming environment being similar to your peers?
I think we should all agree on a definition of developer first. Not just in the sense of "HTML isn't a language", but picking a specific domain in which to compare developers.
I don't think the concept of productivity can span multiple fields. Does a C programmer have the same productivity as a Ruby programmer? On the other hand, maybe some part of a Ruby project needs optimizing, so a routine is written in C. Maybe there can be a definition of productivity that crosses languages... but we don't have one yet.
A C programmer may not have the same productivity as a Ruby programmer - but when you need to write a device driver for Unix/Linux or a Windows System service - Ruby is not a decent choice. Productivity needs to be relative to the task at hand and choosing the correct language for the job is part of that task.
In my last startup we used 3 Measures for productivity.
Number of functions/procedures/methods, Number of lines of non-comment code and the number of defects found in the
code. I wrote far more code than anybody else in the
company. I took on all the really hard stuff that other developers had already failed at. My number of defects per line of code and per function/procedure/method was always lower than all the other developers.
So I would choose to politely disagree. You can measure productivity.
Well what would make you more productive vs. another programmer in solving an identical problem with both of you having all the same programming tools available?
1. Familiarity with the library set / language
2. Familiarity with the problem space
3. Familiarity and speed with tools/programming environment
4. Research speed (and research tools/materials)
5. Mental Program structural planning speed
6. Previous Experience with the specific problem
7. Knowledge with time saving programming structures
8. Skill with mathematics and algorithms
9. Computer Hardware & Internet Connections
10. Ability to choose the best tools for the job
11. Learning speed
12. Motivation, energy, ability to concentrate for long periods of time
And that's listing mostly external factors that are somewhat obvious, not some less obvious speed of mental thinking or something similar.
You can define developer productivity as speed to complete an identical project compared to another and the quality/maintainability/readability of the software they produce to match the project requirements.
Developer productivity is a lot like obscenity. And jazz. It may be hard to define. But it doesn't matter. Because I know it when I see it. YMMV. But that's not my problem. I don't have to define precisely what it means to be hit in the face by a rotting fish. I'm pretty sure I'll know it when it happens.
Now back to Terminal and vi so I can be productive again...
Well you could use peer ranking, and see how well the rankings correlate with one another. That's the method they use in expertise studies for subjective fields.
I don't know. A subjective judgement of the results doesn't mean that a study isn't scientific. You just need to be systematic and to be explicit about how you measure.
Part of my point was, though, that nobody should need to do an objective study. :) Because it's pretty frickin obvious to anybody with a brain who works in the field, in my judgment. And it's not limited to our field. It's a human-wide phenomenon.
From the article: "Critical examination of ill-supported assertions takes a lot more time than making the assertions in the first place"
Anyways, it's really an unfalsifiable claim, isn't it? For any study that doesn't show a 10x difference, you could claim that it didn't have any truly "good" programmers.
The burden of proof should be on those who make the claim.
Apologies for going off topic, but I just read the articles on debunking the MMR <-> autism link. I misread your open paragraph to be:
"It's important to note that (as far as I can tell) the author doesn't actually debunk the existence of precognition, only the claim that studies show precognition to exist."
Yes, but quite vacuous. The author can't prove the non-existence of evidence supporting the 10x claim. He was critiquing our folklor-ish belief in this 10x claim, without evidence, and our gross indifference to this fact.
Let my try another tack, proposing: excluding some bottom <10% that are incompetent, and those with less than 5 years of experience, all programmers have exactly the same 1x productivity capability; all apparent levels of difference are really either ramp-up on new technologies or flaws in development processes and tools. <citation of irrelevant study on something else> <citation of myself, citing the aformentioned study>.
... do you now believe in this 1x claim? No? Hmm, perhaps I need to get it quoted more and then you'll believe it?
The author's point is that "quoting it more" and "it seems reasonable" are the only basis for our belief in the 10x claim. But neither are scientifically, empirically relevant. We should follow up with further studies to confirm the effect, and then investigate the cause. But we don't. And because we have not, we don't have any scientific basis for believing this 10x claim.
This isn't a matter of "dismissing all of our research based on it" -- there is nothing to dismiss. There has been no sufficient research.
Oddly, my initial misread is inaccurate. There are far more studies supporting the existance of precognition than supporting this 10x claim. ;-)
Heck, even one given programmer can see his own productivity varying wildly. There are times I could hack away thousands of lines of working code in a single week, and some other times I'd tinker for months getting no actual job done. Overall I managed to ship from time to time, though I blankly admit I failed the most ambitious projects because I was probably not up to the task.
Even programming at my very basic ability level (actually I never considered myself to be a professional programmer * ) I've seen a number of people who simply never get anything done, and are 10x less productive than I am. The sort of people who are the heroes of thedailywtf.
They never do any better than tinkering around; with some luck they may actually implement some simple code if given very precise requirements because they miss the basic impetus, or the necessary comprehension to start by themselves. Maybe in some huge java drones team, they may add some value, I don't know for sure, I've never worked in a team of more than five programmers anyway.
OTOH I've worked with some guys, thumping the keyboard 16 hours a day for months straight, getting working production code out of the frigging door every single day. So I dare saying I've seen it all : the 10x programmer, the 1x programmer (myself and most others), and the 0.1x programmer (I don't know how many).
All of this sure isn't scientific fact, but simply what I witnessed myself in the past 25 years. It's enough information to guide me through :)
I don't see how it's really debatable that some programmers can be 10x more productive in certain domains. This is especially true in technical domains where it can take years to gain the knowledge required to solve the problem at all. I can think of real-world problems that would require the average programmer several years to solve, yet I know people that could solve them in a day. In fact, almost any specialized task works like that.
I don't claim as extreme productivity differences as the parent but I've seen people do pretty impressive engineering feats and have heard first hand stories of others.
In the cases I've seen what makes people a lot more productive is experience. At Google, Ken and Russ ran circles around me in terms of writing compilers, but I had only written half a dozen toy compilers while someone like Ken has written many more production ones.
My current co-worker Mikko just spent a few weeks re-writing the code to extract a mesh from our level set data. It's a distributed algorithm and our literature search didn't indicate it had ever been published. Starting with someone without domain experience it would be several weeks until the algorithm made sense. It probably would take years to replicate the feat.
Programming is a very specialized endeavor. Some parts of more than others. My own experience indicates that individuals can have magnitudes of difference in their performance when applied to some of these specialized areas.
I would suggest http://norvig.com/spell-correct.html as an example of a very short program that took an expert under an hour to produce which many competent programmers could have spent weeks spinning their wheels on. (With a worse result.)
You will note that the expert is not quicker by virtue of pulling in more sophisticated tools or large libraries. Rather the expert is faster due to knowing how to solve the actual problem.
The very fact there are articles explaining previous solutions to these problem domains is why I just don't believe anyone would ever take a year to get up to speed with something an expert could do in one day.
I can make a great spell checker or a great anti-spam program that takes an expert a day to program in less than a year by just googling it.
There are so few programming domains that aren't fairly transparent where a domain expert has such a massive advantage that a domain novice couldn't gather sufficient knowledge in a year to rival the other programmer's one day effort, given the same level of intelligence, etc.
When it comes to spelling correction, there is a lot of literature on how to do it. So that example I grant you. But could you, before Paul Graham's article, have written an anti-spam program that is that effective? Seriously?
It's not specifically programming, but I've done homework exercises in a night or a weekend as a grad student in mathematics that were main results of PhD dissertations years ago. It's all about point of view. If you have the right way of looking at the problem, things become possible that would look like miracles to someone looking at the problem a different way.
I don't want to belittle your achievement, but the reason will be because your tutors gave you the correct tools which weren't available to your predecessor.
It's certainly not that you're 10x better, it was homework. Your tutors expected you to be able to complete it as they'd spelled it out to you in the lectures.
We're talking about one programmer being 10x better than their peer, not of one generation being more scientifically advanced than the previous.
Given that a) the only way we're measuring programmer competence here is in their speed to complete a task, and b) knowledge is not evenly distributed, how can you tell the difference between a "faster through more knowledge" and an "intrinsically better" programmer?
There are some things that some people can solve in a given time frame, which other people could never solve. For example, could you have ever invented calculus or discovered relativity? What about someone less smart than you? That's an even more extreme example than the one given.
Moving back to programming, debugging specifically, I've had people spend weeks stuck on a bug, only for another programmer to find it in hours or days.
Citation itself often presents as a modality,
with an associated degree of confidence.
I started to wonder if this article was a practical joke. Why was someone writing about programming using the pompous language of literary theory? Was someone trying to pull a Sokal? Then I saw the endnote.
(I realize this is only DH2, but it's such an extreme case that I can't help thinking how lucky we are not to have programming ordinarily discussed in this way.)
Which part of the above did you have trouble with?
"Citation" is the topic being discussed. It's a technical term that refers to how scientific papers refer to other scientific papers.
"Modality" is the concept I've been introducing for three paragraphs. You're supposed to know what it means by then, or I've not been clear enough.
"Present as" is an intransitive verb phrase. I could say the same thing, perhaps more simply, by saying "you can think of a citation as a modality". Perhaps it's this phrase that has thrown you?
[EDIT: I've changed the article to try and make the sentence clearer. Thanks for your feedback.]
It seems to me that "degree of confidence" is clear enough, and that it's clear enough that a modality can have a degree of confidence associated with it.
So the article introduces two technical terms you're perhaps not familiar with, but I'd suppose someone who can read a CS paper or write a program can cope with that much.
It's the word "modality" that throws me. The word by itself is pretty vague -- "somehow related to a mode". The linked Wikipedia article is dense and confusing. You explain it as a way of modifying a statement. Then you say a publication is an "extended modality". That requires another mental stretch. How does the abstract idea of "a publication" connect to modifying a statement? One expects you to explain that in the following sentences, but you don't really. Instead you follow with the seemingly unrelated statement, "A researcher may wish to state a conclusion: “water boils at 100 degrees celsius”. Convention dictates that he should initially add various hedges to his conclusion: ..." So when writing a publication a researcher will customarily add hedges. What does that have to do with a publication being an "extended modality"? Is it really necessary to use such an obscure term? It seems like it shouldn't be this hard to understand what you're saying.
I definitely agree about the linked Wikipedia article on modality. Literary theory is probably not a good way to talk about modality with programmers, but the idea of modality isn't just about linguistics and literature, and isn't as obscure as you think.
A better starting point for programmers and computer scientists would have been modal logic, which uses the modal operators of necessity and possibility.
For example, classical logic uses propositions. I can say "P" in classical logic. In modal logic, I can say "P", "Necessarily P", and "Possibly P", where logical necessity and possibility are modalities. See http://en.wikipedia.org/wiki/Modal_logic -- it's a pretty good overview and it links the logical and epistemological senses of modality.
> You're supposed to know what it means by then, or I've not been clear enough.
I hate to break it to you, but people (especially on a site like HN) have a tendency to skim over text. I hate to repeat myself as much as the next guy, but when you're communicating with human beings it's quite often that you have to beat a dead horse.
The other thing is that the overall sentence isn't necessarily very difficult to read, but the fact that there are so many $10 words gives the impression that it's more difficult to read than it is. So most people (like me) don't try.
With the above noted, I don't know if there's any way it can be improved. Perhaps that's the best way to say it. But those are the reasons others might find it difficult to read.
I find this article to be very clear and readable overall, despite the presence of a few "cultural markers" such as the above language. In comparison with the truly opaque and mostly content-free works of the most notorious poststructuralists, the claim of the article ("assertions about programmer productivity are presented as scientific facts but are not founded on a large body of experimental evidence we expect from such claims") is clearly stated. That said, my own personal experience and observation is that the 10x claim probably underestimates the difference in productivity between the best and worst coders.
Ahem... I really hate to make ad hominem arguments, but when you say things like "In comparison with the truly opaque and mostly content-free works of the most notorious poststructuralists, the claim of the article ... is clearly stated", I have to question if your definition of "clearly stated" is the same as mine. :-)
I'm sure our definitions differ. For comparison, here is a quote from a translation of Guattari's "Chaosmosis" (1992) which I would say is unclear: "The ontological relativity advocated here is inseparable from an enunciative relativity. Knowledge of a Universe (in an astrophysical or axiological sense) is only possible through the mediation of autopoietic machines. A zone of self-belonging needs to exist somewhere for the coming into cognitive existence of any being or any modality of being."
That is the kind of language which made many European intellectuals in the later half of the 20th century notorious. The problem is not just an unusual vocabulary and elaborate sentence structure, but fundamental incoherence and haphazard use of undefined terms. I don't believe the linked article under discussion is really in the same category.
Like others, I'm a little puzzled by this objection to what seemed to me to be a coherent and interesting piece.
I love the self-awareness implied by referencing your own disagreement hierarchy, and humbly wonder whether the following insightful remark may be pertinent here:
“Someone who has a chip on their shoulder about some topic might be offended by a tone that to other readers seemed neutral.”
The author has bad style, but he's not writing nonsense. He edited the statement because he was bullied into it, not because it was nonsense.
I would try to avoid words such as "modality"---"qualification" might be better---but the point he makes is clear. He spends several paragraphs defining what he's talking about and then makes a case. It actually follows the lemma, lemma, theorem formula closely. He goes on to offer evidence for his point. Summarized, he says: over time citations can lend credibility to an otherwise unproved statement. He then says programmer productivity disparities haven't been conclusively demonstrated. That might have been a better starting point for the article.
He's guilty of disorganized writing, and being influenced by literary theory, but his point is otherwise legitimate and capable of being proved or disproved. (I don't know how much he's edited the article since.)
If I understand the author correctly, "modality" might usefully be replaced by something like confidence indicator or confidence signal (in which case the last part of the sentence quoted by pg could be omitted).
My personal opinion is that the big gains in programming productivity in large projects have have to do with how code is organized and structured. The right abstractions are also key to keep the inherent complexity under control. It is not about implementing x feature in t time. It is more about implementing the feature in a coherent way, and the gain is long term. To measure and compare that, you would need to track 2 projects implementing the same features over a period of time. That experiment is hard to do.
I suspect the 10X observation is due in large part to the fact that some programmers are more experienced. I'm much more productive than I used to be, because I've been doing it for 10 years. I've already had to think about a lot of problems, which means I already know solutions for a lot of things. It doesn't mean I'm inherently better than I was, just more experienced.
I expect that this 10X figure is also applicable to things besides software engineering. I'd guess that auto mechanics with 30 yrs of experience running their own auto shop are 10X more productive at solving problems than new guy at the dealership. (Note: solving problems, not just replacing the part. Knowing what part to replace) The Ph.D. who's been researching for 30 years probably is a lot faster at finding solutions than the guy who just passed his qualifiers.
The focus was more on the language factor, and some programmers were students while others were pros, and in some they were self-selected; but still, >10x variation in development time among the Java programmers. 30x if you allow all the above differences. The 10x thing gets quoted way more confidently than the amount of study justifies, but it's not just made up.
I've already made this comment elsewhere on this thread, but just for completeness sake...
The problem with reading off a 30x variation (or even >10x) from Norvig's results is that it discounts the possibility that the same person might take different amounts of time for tasks (relative to the others in the group).
For instance, you might have people in the group that have already worked some variation of that problem. Or you might have some that had colds, were distracted, or just do badly on that sort of enumeration problem, but are killer at other problems. Every one of these possible discrepancies cuts away at the expected magnitude of the actual overall productivity difference between programmers, and should be accounted for.
Of course, since that difference is not what was being studied, they didn't set up the experiments to gather the data we'd need to best estimate the real productivity differences, so it's hard to say what we should conclude...I'm sure there are people that are 1/30th as productive as the best programmers, but what we'd really want to know is how typical they are, and what the overall distribution looks like. That's far less clear.
The 10x productivity claim is one of the unfortunate ones, because anyone with a few years of programming experience can relate to the idea, but it's very difficult to define an objective hypothesis that can be tested quantitatively in any meaningful way.
First you have to define what things like "productivity" are for programmers, and to my knowledge no-one has really done a robust job of even that yet.
Then you have to establish who the "best" and "worst" cases are. After all, I suspect many of us would agree that some programmers make a net negative contribution to their projects, taking more time away from more efficient programmers to fix the resulting problems than it would have taken the stronger programmers to do the work themselves in the first place. This may be inevitable, given the learning curve involved in a field as complicated and diverse as programming.
That all said, I will just mention that Glass also covers this subject, with various other citations, in "Facts and Fallacies of Software Engineering". Given that his final fact is "Many researchers advocate rather than investigate", a criticism very much along the lines of this article, it would be interesting to know whether his sources stack up to more robust criticism if anyone has access to them.
Is there "software engineering" engineering at all?
It is more like "social engineering" or "financial engineering". More to do with ingenuity, than engineering as a practice.
Engineering is an application of scientific - mathematical and physical knowledge to create something. If you don't do the Math, you probably are not an engineer. There is little room for subjectivity.
So, I guess most programmers (if not all) are software technicians. They have practical knowledge, but not the proven theory behind it.
So you will have "Fact and folklore in software engineering", until you can synthesise "something-driven development" using knowledge mathematics and science.
Currently you have some ideologies and studies on them.
In control engineering for contrast to solve a problem, for example a controller regulating the flow of fuel in an engine you have the following procedure:
1. You make a model of the open and closed loop system. You know how do the processes look like in derivatives and you can make transfer functions and state-space representations.
2. You correct the system according to specification, adding to the models.
3. Knowing the physical properties of the different parts of the system you are left with some choices of implementation, but they have different trade-off (physical and economic). You talk with the client and make the appropriate choice. :-)
So, you have the heuristics to solve the problems all build on both practice and science and you don't have the engineers split in schools or ideologies :-)
Software engineer is a term that can help with visa applications. Programming feels a lot like engineering, architecture and shop work combined. A guy with a machine shop and an idea can create a machine that becomes a very useful mass market product even if he didn't do any mathematics other than measurement and arithmetic. What an engineer does with lots of math can create a similarly identical product. Some problems you need an engineer's sophistication with mathematics and physics to create (like a bridge), and similarly in programming, some problems in graphics and what not require sophistication in mathematics and physics to even create it or to advance the field.
As the article points out (and I have before), even if it wasn't a myth, there's no interviewing technique to determine who is at what level.
Also, it should be kept in mind that there is a chance that the original study was contaminated by bad programmers. The truly bad - those who don't really know how to write code at all - are going to vastly underperform those of average competence.
If you had a test that screens out basic competency (an ability to handle CS 101 tasks like assignment, for instance) - how wide would the range be?
Certainly based on 1 study from the 60's... we can't make a good judgment. It's not even a coding study! Even if it was a coding study, it doesn't measure how well they work in teams, or how well they adapt to new languages, or the myriad other things that matter in a programmer's career.
.
Here's a question that deserves study - what interview techniques can make productive teams? What productivity difference is there between those good at brain teasers (programming ones or otherwise), and those that are bad at them? What about those given syntax questions? Those who feel like a good cultural fit? General IQ?
I have looked back in the notes from my last startup. I was CTO and I had to spend time doing fund raising, making sales calls, editing documentation and designs, going to customer meetings. I was never full time in development. I wrote 69 times as much code as the least productive developer and 12 times as much as our "Lead Programmer". Based on my experience 10x is a low estimate. I would say that some people are 100x as productive as others in developing software.
Article talks a lot about science, and then provides no actual studies that refute the original claim. That's not how science works. Devise a hypothesis. Develop a test to confirm or deny. Run test.
And then he tears apart the study with strange assumptions. "Well they were debugging, not programming." Because no programmers in the real world spend a significant amount of their time debugging. I'd consider that to be a significant amount of time for a programmers day. Wish I had code complete in front of me, because I know it did do a break-down of developers time. The one that said that most developers spend more time walking than reading software books.
If you're going to refute McConnell or even Peopleware, you've got to do a study. You can't just say "well cubes seem more productive than an office."
He's not refuting "X", he's refuting the claim that "studies have shown X". To do that you don't need to run your own studies and show different results from X — in fact "X" might even be true. But if the existing studies don't exist, or can be shown to be flawed, then it's perfectly valid to simply point out the holes.
Your debugging criticism is somewhat valid, but the context here in that the original Sackman paper was mostly testing the difference between "online" and "offline" programming (i.e. programming whilst sitting at a computer or not — nothing to do with the internet!): see, for example, http://dustyvolumes.com/archives/497 and http://dustyvolumes.com/archives/500
The research was on whether debugging whilst sitting at a computer would make you more lazy, which is an interesting experiment for its time, but isn't really that connected to the modern concepts of debugging vs programming.
But to me that's like saying no study has proven global warming is real. Because study X only measured ice cores in Antarctica. And study Y only measured tree rings in the amazon.
A far better refutation would be be: Well I ran study Z, and it shows...
(Not that I think the article's claims are as outlandish ad Global Warming Deniers, or that developer productivity has been studied nearly as much as global warming.)
Article talks a lot about science, and then provides no actual studies that refute the original claim. That's not how science works. Devise a hypothesis. Develop a test to confirm or deny. Run test.
To the contrary, a very important part of science is a lot of lazy onlookers scouring what others have done and shouting "You didn't prove what you think you proved!"
Granted, it's always better when they are able to do the experiment right, but failing that, knowing that the original one was done wrong is still extremely valuable information, especially when it's a result so widely quoted.
"Well they were debugging, not programming." Because no programmers in the real world spend a significant amount of their time debugging. I'd consider that to be a significant amount of time for a programmers day.
To me, the problem with taking a study on a single round of debugging as meaningful is that debugging, overall, is a higher variance activity than writing fresh code.
Say there's a large group of exactly identical programmers, who would each take an average of 50 minutes to find a bug, but the amount of time for any individual to find a particular bug ranges uniformly from 10 to 90 minutes. If you give each of them a single bug to find, you're almost definitely going to measure "productivity ratios" that come close to the limit of 9x (with a big enough group). But all of these programmers are identical, so we damn well better not be publishing that as a result that "Programmer productivity varies at least as much as 9 to 1". We need to run more tests, or at least otherwise parse the data in some way in order to figure out what we can really conclude.
In comparison, the act of writing fresh code will usually have lower variance, because it's less of a "search through unfamiliar shit" task, so we could probably take inferences based on single trials there a tiny bit more seriously, though it still doesn't justify sweeping this issue under the rug.
I haven't read the study that we're referring to, so maybe I'm wrong, but since I know that this claim wasn't the primary focus of the paper (they apparently just noticed high internal variation within each of the two groups that they were comparing), I have a sneaking suspicion that we are talking about a single debugging task, and directly comparing the best and worst performers on that task.
FWIW, Norvig's results (http://norvig.com/java-lisp.html) are more meaningful as far as drawing distinctions between programming language productivity (though they don't distinguish whether the languages used are a cause or an effect of productivity differences), but they also cannot be directly used to prove that, for instance, some programmers are 30x more productive than others, because the variance within an individual's productivity is not accounted for.
The example chosen for the first half of this blog post was rather annoying: water boils at 100C by definition, and thus there has never been any scientific debate about this.
I would love to know how a study could be conducted in such a way to prove or disprove this notion. Any ideas? Any existing data sets that could reliably mined for info?
I wouldn't start with a data set. I would start by stating clearly enough the question we are looking for an answer to.
Doing that might involve getting clear on the terms. "Productivity" is too abstract and ambiguous to be used as-is, so we first need to say "here is what we are going to measure".
I believe that what people try to determine is not a very precise measure of productivity, but the fact that productivity measurement doesn't matter so much.
The central message that I remember from Peopleware is that software teams are more about people than about numbers.
Performance is going to vary, often wildly (often for the same people at different times). I loved the anecdote of the woman that wasn't very productive herself, but was a "catalyst", maybe simply a fancy way to say she was fun and nice to work with.
Unless you are Google and have the right process to detect the best and resources to attract them, your best bet is to treat people well and apply common sense... as opposed to apply hard metrics and put pressure, demoralizing your people.
First you would have to define what you are measuring and develop the experiment. Then get enough HN readers to sign up to take the experiment so that you get statistically significant results.
For caned results that you could look at I would think ICFP programming contest or similar would have numbers though a quick look hasn't turned anything up.
This is one of those things that's yet another example of something being treated academicaly which does not need to be treated that way. And of making something much more complex than needed. In real life, in actual practice, any working software engineer or heck even open source hobbyist programmer can tell you that talent and ability and productivity vary significantly between individuals. And this isn't just true among programmers, it's true in other areas/fields as well. This is not controversial, or at least it shouldn't be. And yes, some individuals are clearly 10x or 20x or whatever better than others -- for whatever reason, and the reason(s) doesn't matter too much honestly. It just is.
It's nominally true because many programmers have zero or lower productivity, sometimes due to no fault of their own. If you work on a project for three years, and then it gets canceled, you could say your work had no economic value because it never got in front of customers. If some PHP hack manages to pound something out in a month that brings in $100,000 of business, he looks like a hero, no matter how bad the code is.
In my view, superproductivity is about alignment with your environment. If you work for 12 months a year, but spend 3 of them on side projects that go nowhere, spend 3 months being a sysadmin, and then waste another 5 months on reworking a GUI, there goes about 90% of your productivity.
The "Mythical Man Month" says that about 20% of the time on a software project is spend coding, the rest is requirements work, testing and that kind of stuff. A superprogrammer with supertools might be able to do that in time that approaches zero asymptotically, but unless you can get rid of the other 80%, project cost and time don't get much worse, even in comparison with a coder who is half as productive as average.
The moral? If you want to look like a genius, find some place where (i) requirement work takes little time (and wastes little time downstream with changes) and (ii) testing, deployment and all that is minimal.