Guessing based on uncertain data seems better than guessing randomly? Also if the signal to noise ratio is low my data is likely just noise, but at the same time the decision doesn't matter much. The clearer the difference the more important the decision and the more likely my data is right.

Go take a look at some solutions to the multi-armed bandit problem. A common trait is assigning value (ie. monetary) to each hypothesis. A weaker positive-outcome hypothesis is worth less. The method takes that into account.