hmm have i missed anything?

Any program on your computer can just run "sudo" to escalate itself.

The problem is not the passwordless sudo but running untrusted programs on your computer under your user. They don’t need sudo to steal your SSH keys or inject malicious code in your .bashrc.