Run without swap or with very little swap. I'm serious. Your modern server has enough memory, way more than 4MB, and you care about consistent latency don't you? Also, swap is wearing our your SSD.
It will happen even without swap. Code pages will get evicted and everything will slow down to a crawl. I have magic SysRq enabled so I can invoke OOM killer from keyboard if that happens.