Your observation is a great one, and I'd love to see more data on this as well.
A related point, though, also rings true. Soft skills like being a good coach and effective listening are so underinvested in, that even marginal improvements in those skills lead to huge differences in success.
I see this in engineering leadership workshops that I've run with Jean Hsu and Diana Berlin, where even teaching a handful of coaching and listening skills can have a transformative impact on participants.
At Quip, we run one coding interview that happens on a laptop, and where your conversation and discussions with the interviewer, including how you handle suggestions and feedback, matter a huge deal.
For experienced hires, we'll do deep dives on technical projects that they've worked on. Sometimes, I'll frame these as "Suppose I'm a new member joining that team. Bring me up to speed." These interviews focus on whether the candidate can clearly articulate concepts, explain the big-picture motivations, defend decisions they've made, understand complex technical problems, and stay humble and share lessons learned.
For manager interviews, we'll also do interviews that are one-on-ones with engineers on actual issues that they're facing.
> we'll also do interviews that are one-on-ones with engineers on actual issues that they're facing.
Be careful with this approach. If you're not paying candidates for their interview time, you're not allowed to use their work. Big companies go to great lengths to demonstrate that the entire interview is for the purpose of a hiring decision and nothing more. This is to limit liability. Your approach is very dangerous for your company and if an unhired candidate's idea shows up in your product, even if you arrived at that result independent of the interview, the candidate has a strong case against you in a lawsuit.
If you're doing interviews like this, be sure you've discussed all the nuances with your company's lawyers.
A related point, though, also rings true. Soft skills like being a good coach and effective listening are so underinvested in, that even marginal improvements in those skills lead to huge differences in success.
I see this in engineering leadership workshops that I've run with Jean Hsu and Diana Berlin, where even teaching a handful of coaching and listening skills can have a transformative impact on participants.
If you're interested in future workshops, you can sign up to hear about them here: https://effectiveengineer.typeform.com/to/cDMeZu