> Because a PhD is useful for a lot more than getting a permanent academic position. My first "proper job" was at a research division of a big corporation. They had ~20 PhDs when I started and expanded to ~100 over 3 years. I don't think any of those people would say their PhD was wasted because they didn't get an academic job at the end of it.
Why do they have so many more PhDs now? Because PhDs are a now a dime a dozen, not because a PhD itself is valuable. The opportunity cost of a PhD is four years of lost income during prime employment age, plus extra student loan debt.
Even with a higher salary (and higher tax burden), it's not clear that you come out on top during your lifetime. If you don't come out on top, it's a waste, unless maybe these years you spent doing your PhD were the best years of your life (probably not).
Because these jobs are researcher jobs, and research is a distinct skill-set than what is learned in the typical undergrad and masters program. Could we train researchers in another, more efficient way? Maybe, but PhD program are the current means of training specialized researchers.
> plus extra student loan debt
For researcher jobs, this will almost never be true. The vast majority of STEM PhD students are fully-funded and given a stipend (although usually small). The dynamics of course change if the student has to support other dependents.
> Even with a higher salary (and higher tax burden), it's not clear that you come out on top during your lifetime
For fields like Computer Science, there are many viable and high-salaried career for non-PhDs, so a PhD will likely lose lifetime money. For other fields like biomedical research, then they will probably profit over a full career. But really, people don't do PhDs for the money, people like their fields, they like the university environment, they like being around other researchers, or they like the independence—all of these things can be valuable beyond lifetime salary.
In engineering possibly not. Often tuition is paid for plus a stipend.
>If you don't come out on top, it's a waste,
I don't know. I didn't get a PhD but I did spend a couple of years getting a Masters. I didn't really use what I specifically researched but, who knows, may have helped me get a couple jobs and I enjoyed the time well enough. Of course, engineering salaries tended to be a lot lower then relative to what many CS people today consider normal so I don't think losing out on a couple years salary was a big deal.
Why do they have so many more PhDs now? Because PhDs are a now a dime a dozen, not because a PhD itself is valuable. The opportunity cost of a PhD is four years of lost income during prime employment age, plus extra student loan debt.
Even with a higher salary (and higher tax burden), it's not clear that you come out on top during your lifetime. If you don't come out on top, it's a waste, unless maybe these years you spent doing your PhD were the best years of your life (probably not).