Can't you block ips after x failed attempts? That would make it costly to scan 65536 ports. In a directed attack that might not matter, but against attackers that scan millions of hosts it might give you some minor peace of mind, but if you bother to configure your sshd you probably aren't a target for those attackers anyway.