This applies to any private information - the less a third party has of your data, the smaller the attack surface through which incompetence, malice, and coincidence can burn you.
We deserve privacy because data bout you can be used against you by malevolent actors, whether government or non-government. It's not about whether you have done something wrong or not. The data about you can be used against you.
