If anything those are good reasons to pursue increasing T levels without those side effects. This may be possible if the cause is environmental. The speed with which testosterone levels have been dropping in men seems to indicate it is not natural selection.

There are plenty of people who would trade feeling better for a longer shitty life.