Yes, that's generally true. Galaxies that are redshifted are moving very fast relative to us. But the only reason for them to be moving so fast is due to the expansion of the universe, and these galaxies being sufficiently far away.

The large white elliptical galaxies in the center of the image are in the "foreground", while the orange-ish galaxies are much farther away in the background. This is why the light from the more distant galaxies is curved and distorted by the foreground objects, creating the lensing effect that we see.