And regardless of whether we can infer anything about the true pay inequity from the sample, the response by management described here is just totally awful, and strongly implies behavior that is illegal in the US. You're not allowed to penalize an employee for discussing wages simply because their discussion is factually inaccurate or statistically unsound.

If someone came to me and said "I did a self-selecting self-reporting survey of salaries, and it says I'm underpaid", I think I'd be more likely to fire them for incompetence than to give them a raise.

OK, except that's not what's happening here. What seems to be happening here is someone saying "I did a self-reporting survey of salaries, and there's an existence proof that someone at my level is getting paid $X, can I get paid closer to $X than my current salary." Which is sound.

The statistics and pivot tables are very interesting, but I suspect they're not directly being used for negotiations. Nobody has any good information about how much money they should be asking for. The data this spreadsheet provides is that, if you ask for $X, it's within the range of reason, and not so high that you'll get laughed at.

If I apply for Google and try to negotiate a $200K salary, is that low? high? so high I'll get my offer revoked? (I've heard of that happening in tech!) I have no idea. Knowing even one person who works in my future role and makes $200K would be valuable information. So would knowing that I can't find anyone in that role making more than $110K. Knowing the mean and standard deviation of salaries at 95% confidence is not particularly more useful to me.

It's sound modulo the assumption that everybody is being honest when they report their salaries, sure. Of course, in a company Google's size, there are always going to be outliers, so "someone at my level is getting paid $X" is a long way from "everybody at my level should be getting paid $X".

If I apply for Google and try to negotiate a $200K salary, is that low? high? so high I'll get my offer revoked? (I've heard of that happening in tech!) I have no idea.

Right, and that's the proper use of such anecdata -- to help you formulate your bargaining strategy, not as a bargaining chip itself.

Ironically, if information like this had been available back in 2006, I might be working for Google: A few months after I rejected their offer, I was told that it was "pathetically low" and that I could have negotiated at least 50% more. If I had been offered that much more it might have factored into my decision; but I didn't know I could negotiate, so I ended up simply rejecting the offer.

This is shocking to me. Was it your first formal job offer?

Even if besides them being wrong in this one thing, they would actually be competent and underpaid?

If the answer to the above question is no, why do you even have to elaborate. Then it doesn't actually matter at all if the employee complains rightly or wrongly! I have to conclude the answer has to be yes, then.

It's perfectly understandable that you, as a manager or employer, don't want to pay people for their merit if you don't have to. But it's not something you should be proud of.

I expect developers to be familiar with basic statistics.

Firstly, this isn't a job interview. If someone's been working with you for long enough that they want to discuss a salary adjustment, then shouldn't your assessment of the value that they bring to the table be more informed than looking for "red flags" on the spot?

Secondly, developers are generally not familiar with basic statistics. If it's important to your business that they be better-than-average at stats, then budget some time and money for training.

Thirdly, in the real world, we make decisions from incomplete and biased information all the time. Sometimes, those decisions turn out to be wrong, but on the whole, it still works out better than pretending that we have no information at all. If you're in management and you have access to more complete information, then the smart thing for an employee to do is to have a conversation with you. It makes no sense to fire someone for that.

I have a lot of respect for your technical work, but the management style you're describing so far sounds authoritarian and abusive. I really hope that's not how you'd actually treat your reports, in practice.

I don't know; I've never had to fire anyone yet.

I'll admit that when writing at 1AM I may have let some hyperbole slip into my comments. It's quite possible that my reaction would be "great, your assignment for the next week is to do a rigorous statistical analysis of this data".

On the other hand, as someone else commented, I said "more likely"; and that's probably true. I can't imagine a situation where I'd accept a naive analysis of self-selected self-reporting data as having any substance at all, so I would certainly not give someone a raise based on that.

discuss a salary adjustment

Discussing salary is fine. My point is that bad statistics is worse than nothing in this context.

Actually, even this point is wrong, IMHO. I can easily imagine situations where you get just one data point and correctly infer from it that you're underpaid.

One example is that someone who obviously less competent than you will reveal to you higher salary than you have.

Another example is that someone has access to information will tell you that your salary is low, and you have good reason to trust that person.

I understand that you as an employer naturally don't think that you are or ever will be biased in this way. But it still may happen in the wild.

In any case, I think it was already said, if you still think that bad statistics is worse than none, you should make the information publicly available. It's the similar thing as with rumors.

One data point isn't even bad statistics. It's anecdote. Maybe the other person is being vastly overpaid.

Maybe; but if you think like that, then no sample can give you that information. Because you won't get any absolute reference point either way. In any case, such situation is a good reason to talk about your salary.

So your argument is wrong; the sample size is not important (that doesn't mean it's useless, though!). Either you have some reference point, and then you can judge the fairness of the salary (even from one data point), or you don't have a reference point, and then no sample size will help you get it.

If you get told that a large representative sample of people all get paid far more than you, it suggests that you're the anomaly.

Either you assume a prior distribution of salaries or not. If you don't, then you don't know mean salary, and so group of any size will not tell you if you're anomaly. Because they can all be anomalous as well.

On the other hand, if you assume a prior distribution (which is pretty much what Bayesian statistics does), then even one sample will modify the prior, and you gain information (i.e. "suggestion that you're anomaly or not").

Of course more samples is always better, but if you can make conclusion from multiple samples, then you can make conclusion from one sample.

It seems to me that in the first case, you're saying we cannot assume any prior, but in the second case, you're doing exactly that - assuming that there is a mean - 1st moment of the prior distribution and maybe even other moments - which indicate whether an observation is anomaly or not.

"Hey Coliin, I've been chatting with a couple of people, and I've got a concern about my salary"

"You're fired"

The End

So Colin spoke the truth (as far as I can tell). It is very hard to change a bad culture unless the person at the top is committed to it. I fought a number of injustices and was kicked in the balls many times (metaphorically of course). You learn what you can and can't get away with or end up on the outer - truth be damned. I was often respected for my principles but seen as foolhardy for my commitment to truth. From my experience, gender and race is no basis for judging decent & compassionate management either. I worked with some brutal men and women over the years. It doesn't take Einstein to sort the Wheat from the chaff, people normally knew - although terms like "makes stuff happen", "doesn't take prisoners", "hard ass" were some alternate names for these brutal types. I can't speak for all work environments, but I've seen hundreds of salaries and appraisals over the years. I can not guarantee my experience reflects the broader world of business.

The statement was 'more likely to fire than give a raise'. This does not mean that these are the only two options available. 'More likely' just means the chance is higher overall, whether it's 99% likely to fire or 2% likely to fire.

It's also not what was said - it wasn't "chatting led to a concern", it was "self-reported, self-selecting study says X". That stuff really is a red flag to folks who have a decent grounding in statistics.

I was actually referencing your underlying understanding for your ridicule, not the joke itself.

What might something like this look like?

In this particular context, the biggest threats to data integrity are probably:

1. People lie, and men probably lie about their salaries more than women,

2. Willingness to divulge salary is highly culture-dependent, and in societies where salary is seen as a sign of social success, can be highly salary-dependent (and probably more so for men than for women).

If you're interested in learning more I suggest looking for information about how national census data is processed; they have lots of experience dealing with these sorts of issues (albeit with different factors influencing response rates and data reliability).

Certainly it's manager discretion, but that's usually the right thing to do.

(There are other forms of bonusing that would make more sense in such a case, like a spot bonus, or whatever)

Perhaps, if you ever end up managing people, you'll hopefully take a step back and contemplate investigating whether there was, in fact, a valid concern.

Of course, very few would dare to do that because they know that the data will be hiding all sorts of embarrassing and possibly resignation-causing truths. Which is kind of shitty if you think about it, especially when taken with the notion that "companies are people"... sociopathic, lying, cheating people...

For one thing, fairness is subjective. For many, "equal pay for equal work" is fair, but many others believe that seniority, or a better CV, or many others factors should be rewarded differently.

Secondly, there are issues of perception. A worker might think (s)he should be paid more because (s)he always leaves later than his/her colleague, without taking into account that the colleague prefers to take shorter lunches or fewer breaks and leave earlier. You can explain it to them, but why would they believe you?

It'd be different if a company published the salaries from the start, but switching after growing to a huge size can be very disruptive, and I don't blame managers for avoiding opening that can of worms.

(Again, I don't believe the salaries at Google are perfectly fair, I'm talking about an hypothetical company)

What sampling biases do you think might be present in this sampling that invalidate the stated conclusions?

For example, for a conclusion about gender differences to be exaggerated by the data there would have to be systematic biases to sampling women with lower salaries, and men with higher salaries, in the same position. Does this seem likely?

For a self-selected sample, absolutely. Self-worth is far more tied to salary in males than in females. (I don't think this is innate; I think it's a result of social conditioning which tells males that their role in society is to be breadwinners.)

But you don't need to be able to point to a specific source of bias to say that self-selected samples are problematic. It's up to the person who intends to use the statistics to show that they were either gathered reliably or carefully examined to remove biases.

I certainly agree that if someone is trying to make a statistical claim then they should be the ones to do the due diligence on the data, and on their analyses, to show that they stand up - I would certainly expect this in a more formal publication.

What I am disputing is that one can dismiss the conclusions simply due to the lack of these checks, without considering what the data might tell you.

Nearly always such dismissive arguments strike me as a disagreement with the conclusions, rather than a claim about failings in the data - a more convincing argument would present a reason why the data are flawed, such as the one you have given in this response (even if I don't understand it).