This is my observations as well. For 20+ years I have run ssh on a high port, with exception to my sftp server. The sftp server is hit every day, all day. I have received 0 hits to my ssh port on all my other servers. Even if they hit that port, they would not see anything, as I use a poor-maps port knocking using iptables string matching, but I would still see the attempts in the iptables counters and they are always 0.

FWIW, when I chose my port, I looked at port scanning statistics back in the day, looking for the least scanned ports. It appears those stats have held true for a couple decades at least.