If the speed limits were set to the safest speed to drive at you would be right. They aren't though. They are, as the original post says, set arbitrarily.
Often they are set lower than the safest speed to be driving on that road (generally the speed that the rest of the traffic is traveling at). So by not speeding you increase the probability that you screw up and do something bad.
In California at least - the speed limit cannot be different (more than some statistical measure) of the real world speed people actually drive on the road. Theres a process for requesting a survey of actual vehicle speed for a street. If the measures speeds differ in a statistically significant way from the posted speed limit, the posted speed limit has to be changed.
Often they are set lower than the safest speed to be driving on that road (generally the speed that the rest of the traffic is traveling at). So by not speeding you increase the probability that you screw up and do something bad.