Adding a little context for those not familiar with orbital mechanics: the speed of a circular orbit is directly related to its distance from the barycenter (i.e. the Sun). The asteroid belt is between the orbits of Mars (orbiting at 24km/s) and Jupiter (13km/s). Whilst an asteroid that strays far enough to hit Earth (orbiting at 30km/s) is by definition not in an exactly circular orbit, nor one always between Mars and Jupiter, the difference in speeds isn't that great. That accounts for ddahlen's point about the limited range of velocities.

A very rough calculation of mine involving a hypothetical asteroid in a elliptical orbit extending as far as Jupiter and right down to Earth, assuming no difference in orbital inclination to Earth and no significant gravitational perturbations, would result in a relative speed of 5km/s. The actual impact speed would be greater due to Earth's own gravity, adding an extra 11km/s.

Not all asteroids are from the asteroid belt, but I am under the impression that visitors from the outer solar system (which could be as fast as the upper bound that ddahlen mentions) are much more infrequent than stray asteroid belt objects, so the median impact speed would still be relatively slow.

Assuming a prograde orbit. Retrograde asteroids are uncommon, but we know of over 100.

Fun facts:

  Earth's orbital diameter is:      ~30 x 10^7 km
  number of seconds in a year is:   ~ π x 10^7 s
  so, Earth's orbital velocity is:  ~30        km/s

  speed of light, c, is:            ~30 x 10^4 km/s
  so, Earth's orbital diameter is:  ~ 1,000    light seconds
  and, Earth's orbital velocity is: ~ 0.0001   c

Ahh escape velocity hadn't thought of that, thanks. So it would take something like Oumuamua to be much faster?

We have spotted a grand total of 2 interstellar objects, they were moving faster, but are many orders of magnitude less numerous then the local stuff.

Just doing some back of the envelope calculations, looks like Omuamua was moving about 165,000km/hr (relative to Earth) when it was about at Earths orbital distance.

This speed is not actually a crazy number, it is a lot faster than the majority of things which could hit us, but there are geometries of things in our solar system which can reach these relative velocities. (For example things in retrograde, IE: reverse orbits) can lead to basically escape velocity + earths velocity.