The counterpoint to this is that if you're willing to accept patching upstream as an option to make "key workloads" (read: partial benchmarks) perform better, what's stopping you from adding the necessary annotations instead?

The answer is basically that you were never going to, you're just externalizing the cost of making it look like your compiler generates faster code.