I think we split based on gender and race because those are clear discriminators. Height or hairstyle are more fuzzy, and probably biologically less relevants to adapt to the environment.
When I said clear discriminator I mean they are binary: either make or female, either black or white. Height or attractiveness are more diffuse for some people you can be tall/attractive but not for others. Gender and race are clear.
In my opinion height or attractiveness play a role in the society (a big role), but it is easier to talk about gender or race because the clear distinction, bigger represented groups and historical reasons.
Probably height is more relevant for leadership than gender: http://journals.sagepub.com/doi/abs/10.1177/1368430212437211