I think we should all agree on a definition of developer first. Not just in the sense of "HTML isn't a language", but picking a specific domain in which to compare developers.
I don't think the concept of productivity can span multiple fields. Does a C programmer have the same productivity as a Ruby programmer? On the other hand, maybe some part of a Ruby project needs optimizing, so a routine is written in C. Maybe there can be a definition of productivity that crosses languages... but we don't have one yet.
A C programmer may not have the same productivity as a Ruby programmer - but when you need to write a device driver for Unix/Linux or a Windows System service - Ruby is not a decent choice. Productivity needs to be relative to the task at hand and choosing the correct language for the job is part of that task.
In my last startup we used 3 Measures for productivity.
Number of functions/procedures/methods, Number of lines of non-comment code and the number of defects found in the
code. I wrote far more code than anybody else in the
company. I took on all the really hard stuff that other developers had already failed at. My number of defects per line of code and per function/procedure/method was always lower than all the other developers.
So I would choose to politely disagree. You can measure productivity.
Well what would make you more productive vs. another programmer in solving an identical problem with both of you having all the same programming tools available?
1. Familiarity with the library set / language
2. Familiarity with the problem space
3. Familiarity and speed with tools/programming environment
4. Research speed (and research tools/materials)
5. Mental Program structural planning speed
6. Previous Experience with the specific problem
7. Knowledge with time saving programming structures
8. Skill with mathematics and algorithms
9. Computer Hardware & Internet Connections
10. Ability to choose the best tools for the job
11. Learning speed
12. Motivation, energy, ability to concentrate for long periods of time
And that's listing mostly external factors that are somewhat obvious, not some less obvious speed of mental thinking or something similar.
You can define developer productivity as speed to complete an identical project compared to another and the quality/maintainability/readability of the software they produce to match the project requirements.
Developer productivity is a lot like obscenity. And jazz. It may be hard to define. But it doesn't matter. Because I know it when I see it. YMMV. But that's not my problem. I don't have to define precisely what it means to be hit in the face by a rotting fish. I'm pretty sure I'll know it when it happens.
Now back to Terminal and vi so I can be productive again...
Well you could use peer ranking, and see how well the rankings correlate with one another. That's the method they use in expertise studies for subjective fields.
I don't know. A subjective judgement of the results doesn't mean that a study isn't scientific. You just need to be systematic and to be explicit about how you measure.
Part of my point was, though, that nobody should need to do an objective study. :) Because it's pretty frickin obvious to anybody with a brain who works in the field, in my judgment. And it's not limited to our field. It's a human-wide phenomenon.