LLMs predict the next token. Imagine someone said to you, "it takes a musician 10 minutes to play a song, how long will it take for 5 musicians to play? I will work through the problem step by step".

What are they more likely to say next? The reasoning behind their answer? Or a number of minutes?

People rarely say, "let me describe my reasoning step by step. The answer is 10 minutes".