My VPN/Fileserver VPS is not behind Cloudflare, and I haven't had any trouble for years. Only the SSH port is accessible from outside (which is probably not even necessary), with password login disabled. I use fail2ban and a few other extra layers of security.