It would have to go down far enough to make the refi worth it, but in expensive markets that doesn't take much. When rates bottomed out during Covid the RE bankers I knew were swamped with refi work, I think even on recent stuff already at 4% to 5%. It was like a perfect storm: low rates -> bidding wars driving up prices -> refi to a lower rate, fees paid themselves back in a hurry, take out some equity to do other home-office-upgrades and similar too.

But yes, you should never assume you're gonna be able to refi. Not that that stops everyone. :(