I've done over 1000 interviews and my experience agrees with their findings that talking about a project is not a good predictor of coding. I usually left detailed resume questions for the end since so many candidates would bomb the coding part of the interview.
I'm surprised they didn't get stronger results from fizz buzz, but I noticed among the candidates I saw that the percentage of 'non-coders' is substantial but not a majority.
One thing missing from this investigation is a measure of solution quality. A good portion of candidates who actually finished coding questions with me ended without thoroughly understanding how their code worked and/or had code that would be hard to maintain. Other candidates would write top-notch code but were unable to explain their thought process to some extent. These are critical pieces to the interview that contribute much more 'color' than 'score' and are important to note.
I really don't mean to cast aspersions on your interviewing abilities, but the common factor in all of the interviews you've conducted is you, and how you talk to a candidate abut prior projects can really matter. At one extreme you have "tell me about your last project" free-form. At the other you have highly-specific drilling down on a relevant technical issue (e.g. flow-control mechanisms in Ethernet networks) or behavioral characteristic (e.g. openness to collaboration). Which of these have you tried? Have you observed any difference in effectiveness between them? My own experience, from interviewing people who conduct interviews themselves, is that people who say "asking about projects doesn't work" just don't know how to do it right. Again, I'm not saying that applies to you personally, but what information can you provide to refute the observation?
I totally agree with you that the "ask about projects" question requires care. And, yes, I did see differing results depending on how the discussion went. When I did the question, I've run it as both a 1-minute quickie and a 15-minute conversation. I tried my best to dig an solid technical exposition out of the candidate if I could.
Typical cases where the project discussion was not helpful:
* Senior engineers could often talk in great detail about things they'd built, but then would most often bomb coding questions.
* A lot of MS/PhD-level new grads could give great talks (about almost anything) but again would bomb coding stuff.
* Lots of ESL candidates struggled tremendously to convey basic project details, but nevertheless were solid coders. From my TAing experience in grad school, I noticed that there are a lot of ESL CS students who have poor oral communication skills but are orders of magnitude stronger at written communication. (This made strong textbooks and written references crucial to the courses I TAed).
* Sadly we hired quiet a few senior / PhD-level people who could give great talks, were highly responsible, but were eventually fired for poor coding. (There were also some management disasters associated with those cases, but stronger coding skills would have helped them nevertheless).
Interviewing has both subjective and 'objective' parts. The most objective data one can collect are things like completion times, the actual code written, and (perhaps) raw contextual data like a measure of the candidate's mood (were they sweating like mad?). Project discussions rarely recover solid objective data. So, I'd definitely recommend against outsourcing those questions to a service like Triplebyte, and junior employees should probably also not be spending 15 mins on projects for /every/ candidate.
The only time I've found talking about a project to be a good predictor is when it's a type of project or tech stack that I'm familiar with. In that case, I can ask very specific probing questions about problems I'd encountered with that problem or tech. In answering these questions whether and how often they talk about specific solutions they came up with versus saying something along the lines of "well I dunno, I wasn't responsible for that piece" or "I'm not sure about the details, I was more involved with the higher level decision making" can be very telling.
I'm surprised they didn't get stronger results from fizz buzz, but I noticed among the candidates I saw that the percentage of 'non-coders' is substantial but not a majority.
One thing missing from this investigation is a measure of solution quality. A good portion of candidates who actually finished coding questions with me ended without thoroughly understanding how their code worked and/or had code that would be hard to maintain. Other candidates would write top-notch code but were unable to explain their thought process to some extent. These are critical pieces to the interview that contribute much more 'color' than 'score' and are important to note.