I was just playing around with this problem. I ended up firewalling the SSH port for all but my personal IP, then have wireguard set up so I can use it from within my wireguard network. Works perfectly so far as long as I have my clients set up.