I kind of get the idea: if damage is coming to one side of the body, then the sensory data reporting it would be better handled by putting the brain half dealing with it on the opposite side, to give it a chance to react rather then also be damaged.

But, I think the visual processing explanation is more compelling - since it seems unlikely this scenario would arrive in enough non-catastrophic situations to be evolutionary significant.