Are you sure it's always 5? On my playthrough I had no trouble growing to around 10. Didn't feel to be specifically limited to 5.