That's why podcasts are so much better on 2-2.4x speed with a silence skipper. I get an effective 3x speed and as long as I'm paying attention I still get every word. It sounds like gibberish to most newcomers but if you slowly speed up you don't even notice the increments.
Lectures may require a slower speed. I am talking about more general podcasts where there is more of a discussion going on. If the speaker is fast or if there is a wealth of numbers being recited you may need to slow it down. I don't have a single podcast that I run slower than 2x which yields an actual 2.6x with silence skipping.