Developer productivity is a lot like obscenity. And jazz. It may be hard to define. But it doesn't matter. Because I know it when I see it. YMMV. But that's not my problem. I don't have to define precisely what it means to be hit in the face by a rotting fish. I'm pretty sure I'll know it when it happens.
Now back to Terminal and vi so I can be productive again...
Well you could use peer ranking, and see how well the rankings correlate with one another. That's the method they use in expertise studies for subjective fields.
I don't know. A subjective judgement of the results doesn't mean that a study isn't scientific. You just need to be systematic and to be explicit about how you measure.
Part of my point was, though, that nobody should need to do an objective study. :) Because it's pretty frickin obvious to anybody with a brain who works in the field, in my judgment. And it's not limited to our field. It's a human-wide phenomenon.
Now back to Terminal and vi so I can be productive again...