This seems like a non-issue, unless I'm misunderstanding. If failures can be used to help game benchmarks, companies are doing so. They don't need us to avoid compiling such information, which would be helpful to actual users.
People might want to use the same test scenario in the future to see how much the models have improved. We can't do that if the example gets scraped into the training data set.
