I think WASAPI has comparable latency to ASIO drivers. I can get around 20-30ms latency with WASAPI though ASIO takes me down to 10-15. Granted I'm using a dedicated USB audio interface though.

You should be able to easily sustain below 5ms buffer size, and RT_PREEMPT Linux has no problems keeping a few hundred microseconds of buffer filled (a suitable PCIe soundcard should easily make 100us buffer level reliable; that's 9.6 samples at 96kHz sample rate).

Yes, sub-ms latency is hard. But the infrastructure exists.