# A Simple Way to Deal with Cherry-picking

arXiv: https://arxiv.org/abs/1810.04996 · HTML: https://mlsanity.com/paper/1810.04996/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Oct 2018.

## Built on by (most cited first, 1 of 1)

- [How Long short-term memory artificial neural network, synthetic data, and fine-tuning improve the classification of raw EEG data](https://mlsanity.com/paper/2604.04316/) (2026)

## Abstract (arXiv)

Statistical hypothesis testing serves as statistical evidence for scientific innovation. However, if the reported results are intentionally biased, hypothesis testing no longer controls the rate of false discovery. In particular, we study such selection bias in machine learning models where the reporter is motivated to promote an algorithmic innovation. When the number of possible configurations (e.g., datasets) is large, we show that the reporter can falsely report an innovation even if there is no improvement at all. We propose a `post-reporting' solution to this issue where the bias of the reported results is verified by another set of results. The theoretical findings are supported by experimental results with synthetic and real-world datasets.