One of the other comments mentioned matching voice to protagonist, which would fit your example for Harry Potter.
There are likely many factors that go into selecting the right voice here - my main point is the same voice shouldn't be used for all books. It's likely a simple heuristic is better than "male voice for all", though no approach will be perfect without the opinion of the author, which isn't available unfortunately.
Indeed, voice matching to book "by some magic metric" is the way to go.
My comment wasn't entirely clear, it was the "voice matched to author gender" part that prompted a response.
In the great scheme of thing any voice reading aloud is an advance for people that require or like to hear books read out, improvements can come on a per book basis.
The end goal is likely a mix of bespoke readings by gifted voive readers (Fry) and guided "selectable AI voice" readings that can do can do clear and correct pronunciation and pacing with the voice of Jamie Erl Jonas (totally not James Earl Jones), Skarlat Johnson, or that Chipmunk character.
J.KRowling's books, as mentioned, are famously well read by Stephen Fry and Jim Cook, both male voices for an initialed (but female) author.
But what then for J.KRowlings Cormoran Strike series under the pseudonym Robert Galbraith?
Female or male voice for modern crime fiction male detective novels?