Math is a tool for structured reasoning. We often want to reason about things that physically exist. Why should we use a tool that assumes the existence of something that literally cannot physically exist? That introduces the danger that of admitting all sorts of unphysical possibilities into our theories.

I think Baez's paper, Struggles with the Continuum, shows a lot of past difficulties we've had that resulted from this:

https://arxiv.org/abs/1609.01421