Mutexes shouldn't be able to cause priority inversion, there's enough info there to resolve the inversion unless the scheduler doesn't care to - i.e. you know the priority of every thread waiting on it. I guess I don’t know how the Linux scheduler works though.
But it's not safe to do anything with unbounded time on a realtime thread, and malloc takes unbounded time. You should also mlock() any large pieces of memory you're using, or at least touch them first, to avoid swapins.
if you have to wait on a mutex to get access to shared resource (like the book keeping inside your malloc's heap) then you have to wait in order to make progress - and if the thread that's holding it is at a lower priority and is pre-empted by something lower than you but higher than them then you can't make progress (unless your mutex gives the thread holding it a temporary priority boost when a higher priority thread contests for the mutex)
(this is not so much an issue with linux but with your threading library)
I'm completely in agreement that you shouldn't be mallocing, that was kind of my point - if you just got a key change from the cable stream and you can't get it decoded within your small number of millisecond window before the on-the-wire crypto changes you're screwed (I chased one of these once that only happened once a month when you paid your cable bill .....)
> (this is not so much an issue with linux but with your threading library)
If your threading library isn't capable of handling priority inheritance then it's probably Linux's fault for making it not easy enough to do that. This is a serious issue on AMP (aka big.little) processors, if everything has waits on the slow cores with no inheritance then everything will be slow.
But it's not safe to do anything with unbounded time on a realtime thread, and malloc takes unbounded time. You should also mlock() any large pieces of memory you're using, or at least touch them first, to avoid swapins.