Side channels are a huge danger. An example is cryptographic functions have been cracked because of timing differences based on the key or data being encrypted. This is why cryptographic ciphers are implemented in constant time code (i.e. code that always runs in the same amount of time regardless of its input).