I think the more simple and complete solution is to limit multiple characters interactions with one object similar to the way the rules limit one character interacting with multiple objects. Note that even without readied actions, an infinite number of characters could still pass an object in the space of a round, each passing it on their turn, so long as they were arranged in space in initiative order, so limiting readied actions both doesn't solve this (and allowing readied actions to be a bypass to others readied actions opens up as much space for exploitation as it closes.)
The simple solution is just to recognize that the Ready action lets somebody attempt to do a thing, it doesn't mean they automatically succeed.
So N people can certainly declare that they all want to pass an object around, for any value of N. But if the object would need to move at supersonic speeds for them to all succeed, then obviously one of them won't succeed. (And the subsequent people won't do anything because the trigger for their Ready action didn't occur.)
