It's not that much effort either as owned slaves were included on the census.

Pretty much if you were an affluent family that could afford one, you owned at least one slave.

Notably, Mitch McConnell's, Obama's, and Biden's ancestors are all confirmed to have owned slaves. Not surprising as they all come from affluent families.