This is notable, because this is what rurban/smhasher does in the small key test, which many are using nowadays.

In SpeedTest.cpp, there is a function called timehash():

  uint64_t *aligned_result = (uint64_t *) aligned_alloc( 64, 256 / 8 );
  uint64_t *aligned_buf = (uint64_t *) aligned_alloc( 64, 256 );
  memcpy( aligned_buf, key, len );

      begin = rdtsc();
      for (int i = 0; i < repeat; i++) {
        hash(aligned_buf,len,seed,aligned_result);
        seed += aligned_result[0];
        aligned_buf[0] += aligned_result[0];
        aligned_buf[1] += aligned_result[1];
      }
      end = rdtsc();

Take away the aligned_buf += aligned_result, and the additional latency goes away.

When I get a chance, I'll try your method of randomizing input.