> I'm starting to think the latency from the video encoding/decoding could be a bigger issue than the network itself.

It depends on the setup but encoding and decoding combined can be done in less than 7 ms (1 frame at 144 fps) on good hardware:

https://blog.parsecgaming.com/testing-game-streaming-input-l...

Last year on more modest hardware I would regularly see around 12 ms (1 frame at 83 fps).