If your original lender were to refuse you could refinance with a different one. Your lender knows this and therefore will let you refinance to keep your business (I'm actually doing this now).

You do have to pay loan original fees again though which can be 1-2% of the balance of the loan so you have to compute when it actually pays off for you and whether it's worth it.