In undergrad we had an alumni, who had climbed the ranks at Bombardier, come and talk about aircraft and engine evolution. They are 100% trying to make them as cost-effective as possible and fuel economy is a huge driver for that.
The challenge is that not only is there a huge upfront investment into a new unconventional airframe, with the associated risk of failure that you point out, but if you are successful you're now going to be completely retooling and retraining everyone since no one has experience mass producing airframes like this. For that kind of investment... you'd better have a pretty damned good cost or efficiency argument.
Apparently a 747 has a cruise L/D of around 17 and an A340 is around 19. The U-2 is up around 25. There's not a whole room left between what we've got now and gliders as far as L/D goes, so the only place really left to try to improve is the engines themselves... and modern high-bypass turbofans are really marvels of squeezing as much energy as you can out of a (relatively) lightweight piece of machinery. We build the hot bits in those engines out of, literally, magical single crystals of crazy alloys to be able run them as hot as possible to maximize their efficiency.
Just as an example, a lot of companies are using jets on short haul flights where a turboprop would be a lot more fuel efficient. On short trips the speed difference doesn't mean much.
Again, there's some complex calculation about training, fleet diversity, flights per airframe per day etc.
If fuel is cheap, it makes sense to use the more fuel gulping plane because other costs are better. If fuel is expensive, maybe it then makes more sense to use more turboprops even though it means other complications.
Turbofan engine compression ratio and bypass ratio have been increased but we are nowhere near the limits of total aircraft efficiency.
Just as an example, Perlan II glider has glide ratio of 43. Of course, it's for a totally different use case and useless directly as an airline. But as comparison the most efficient airliner, the ATR turboprop has glide ratio of about 15.
The challenge is that not only is there a huge upfront investment into a new unconventional airframe, with the associated risk of failure that you point out, but if you are successful you're now going to be completely retooling and retraining everyone since no one has experience mass producing airframes like this. For that kind of investment... you'd better have a pretty damned good cost or efficiency argument.
Apparently a 747 has a cruise L/D of around 17 and an A340 is around 19. The U-2 is up around 25. There's not a whole room left between what we've got now and gliders as far as L/D goes, so the only place really left to try to improve is the engines themselves... and modern high-bypass turbofans are really marvels of squeezing as much energy as you can out of a (relatively) lightweight piece of machinery. We build the hot bits in those engines out of, literally, magical single crystals of crazy alloys to be able run them as hot as possible to maximize their efficiency.