The populations of both people who go to college and those who do not are so large that I feel like we could likely control for effects (at least extreme ones like you're talking about) and make useful judgments.
While I only pointed to the extremes, even across the general population the problem with trying to compare is that college is setup to filter those who are less able. Be that less able to pay, less able to withstand the academic rigour, less able to take risk, less able to integrate in certain social settings, etc. As a result, only the top (for some definition of top) people are able to enter and finish their academic programs.
It's kind of like looking at the Olympics to see how it has shaped the athletic ability of the athletes compared to the ability of the population who have never been to the Olympics. You are bound to find outliers[1], but observing that Olympic athletes are usually stronger athletes might leave you to believe that going to the Olympics makes one a significantly better athlete. However, in reality, those same people would still be among the best athletes even if they never went anywhere near the Olympics.
Olympic athletes are a tiny population and there is no program, for instance, to let mediocre athletes in, while there are programs to let economically disadvantaged students into college. I think it should be quite possible to control for those effects.
> there are programs to let economically disadvantaged students into college.
But, to stick with the comparison, many countries have programs to help out athletes who are economically disadvantaged as well to allow them to make it to the Olympics, assuming they already have demonstrated sufficient athletic ability to justify the funding. Just as sufficient academic ability is required for any college subsidy I have ever come across.
Although I'm not sure ability to pay ends at economic disadvantage. I don't feel that I can pay for college at this point in my life, with many preexisting financial commitments that need to come first, and I have a developer's salary. Nobody is going to pay my way. Not being poor doesn't mean money is readily available for such luxuries. There is a very real opportunity cost there.
What about academically disadvantaged students? Are there any studies to show how people who couldn't possibly graduate from college if they tried to the best of their ability fair by being given access to the entire course load and awarded a degree even with their technical failure? That would be very interesting and a way to start to control the variables.