>With a human, you can’t really do an A/B test to determine if they would have prioritized a candidate if they hadn’t included some signal; it’s really easy to rationalize away discrimination at the margins.
Which is part of the reason that discrimination doesn't have to be intentional for it to be punishable. This is a concept known as "disparate impact". The Supreme Court has issued decisions[1] that a policy which negatively impacts a protected class and has no justifiable business related reason for existing can be deemed discrimination regardless of the motivations behind that policy.
Justifiable business reason is still a strong bar. For example, with no evidence in either direction for a claim there is no justifiable business reason even if the claim is somewhat intuitive. So if you want to require high-school diplomas because you think people who have them will do the job better you better track that data for years and be prepared to demonstrate it if sued. If you want to use IQ tests because you anticipate smarter people will do the job better you better have IQ tests done on your previous employee population demonstrating the correlation before imposing the requirement.
EDIT: my parent edited and replaced their entire comment, it originally said "you can't use IQ tests even if you prove they lead to better job performance". I leave my original comment below for posterity:
This is not true, IQ tests in the mentioned Griggs v. Duke Power Co. (and similar cases) were rejected as disparate impact specifically because the company provided no evidence they lead to better performance. To quote the majority opinion of Griggs:
> On the record before us, neither the high school completion requirement nor the general intelligence test is shown to bear a demonstrable relationship to successful performance of the jobs for which it was used. Both were adopted, as the Court of Appeals noted, without meaningful study of their relationship to job performance ability.
They said "it’s really easy to rationalize away discrimination at the margins." My reply was pointing out that there is little legal protection in rationalizing away discrimination at the margins because tests for disparate impact require the approach to also stand up holistically which can't easily be rationalized away.
I think perhaps you are looking at a different part of the funnel; disparate impact seems to be around the sort of requirements you are allowed to put in a job description. Like “must have a college degree”.
However the sort of insidious discrimination at the margin I was imagining are things like “equally-good resumes (meets all requirements), but one had a female/stereotypically-black name”. Interpreting resumes is not a science and humans apply judgement to pick which ones feel good, which leaves a lot of room for hidden bias to creep in.
My point was that I think algorithmic processes are more testable for these sorts of bias; do you feel that existing disparate impact regulations are good at catching/preventing this kind of thing? (I’m aware of some large-scale research on name-bias on resumes but it seems hard to do in the context of a single company.)
>disparate impact seems to be around the sort of requirements you are allowed to put in a job description.
That is a common example, but it is much broader than what goes on a job ad. For example, I have heard occasional rumblings about how whiteboard interviews are a hiring practice that would not stand up to these laws (IANAL).
>My point was that I think algorithmic processes are more testable for these sorts of bias
Yes, this is true, but that doesn't really matter. If there is consistent discrimination happening at the margins, that will be evident holistically. If that is evident holistically and there is no justification for it, that is all we need. We don't need to run resumes through an algorithm to show that discrimination is happening at an individual level. We just need to show that a policy negatively impacts a protected group and that the policy is not related to job performance.
>do you feel that existing disparate impact regulations are good at catching/preventing this kind of thing?
I think the bigger problem than the regulations is that there is an inherent bias against these type of cases actually being pursued. First, it is difficult to identify this as an individual so people don't know when it is happening. Additionally, people fear the retribution that would come from pursuing this legally. People don't want to be viewed as a pariah by future employers so they often will simply move on even if their accusations are valid.
Yes, but a holistic test requires a realistic counterfactual. That's the problem. There is no way to evaluate that counterfactual for a human interviewer.
It is true that extreme bias/discrimination will be evident, but smaller bias/discrimination, particularly in an environment where the pool is small (say, black women for engineering roles) is extremely hard to prove for a human interviewer. Your sample size is just going to be too small. On the other hand, if you have an ML algorithm, you can feed it arbitrary amounts of synthetic data, and get precise loadings on protected attributes.
If you ever intent to study law, become involved in a situation dealing with disparate impact, or are at the receiving end of disparate impact, knowing the legal definition may be helpful too. The DoJ spells[1] out the legal definition of disparate impact as so:
ELEMENTS TO ESTABLISH ADVERSE DISPARATE IMPACT UNDER TITLE VI
Identify the specific policy or practice at issue; see Section C.3.a.
Establish adversity/harm; see Section C.3.b.
Establish disparity; see Section C.3.c.
Establish causation; see Section C.3.d.
My point is that by the plain meaning of words you're right, disparate impact means any two groups impacted differently, regardless of anything else. In law, it means that an employment, housing, etc. policy has a disproportionately adverse impact on members of a protected class compared to non-members of that same class. It's much more specific and narrowly defined.
Which is part of the reason that discrimination doesn't have to be intentional for it to be punishable. This is a concept known as "disparate impact". The Supreme Court has issued decisions[1] that a policy which negatively impacts a protected class and has no justifiable business related reason for existing can be deemed discrimination regardless of the motivations behind that policy.
[1] - https://en.wikipedia.org/wiki/Griggs_v._Duke_Power_Co.