# Over-optimism in benchmark studies and the multiplicity of design and analysis options when interpreting their results

arXiv: https://arxiv.org/abs/2106.02447 · HTML: https://mlsanity.com/paper/2106.02447/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Jun 2021.

## Built on by (most cited first, 15 of 15)

- [Deep Learning for Survival Analysis: A Review](https://mlsanity.com/paper/2305.14961/) (2023)
- [YAHPO Gym -- An Efficient Multi-Objective Multi-Fidelity Benchmark for Hyperparameter Optimization](https://mlsanity.com/paper/2109.03670/) (2021)
- [Position: Why We Must Rethink Empirical Research in Machine Learning](https://mlsanity.com/paper/2405.02200/) (2024)
- [Vote'n'Rank: Revision of Benchmarking with Social Choice Theory](https://mlsanity.com/paper/2210.05769/) (2022)
- [From Variability to Stability: Advancing RecSys Benchmarking Practices](https://mlsanity.com/paper/2402.09766/) (2024)
- [On the role of benchmarking data sets and simulations in method comparison studies](https://mlsanity.com/paper/2208.01457/) (2022)
- [A benchmark of categorical encoders for binary classification](https://mlsanity.com/paper/2307.09191/) (2023)
- [Modern Neural Networks for Small Tabular Datasets: The New Default for Field-Scale Digital Soil Mapping?](https://mlsanity.com/paper/2508.09888/) (2025)
- [LimeSoDa: A Dataset Collection for Benchmarking of Machine Learning Regressors in Digital Soil Mapping](https://mlsanity.com/paper/2502.20139/) (2025)
- [Kriging prior Regression: A Case for Kriging-Based Spatial Features with TabPFN in Soil Mapping](https://mlsanity.com/paper/2509.09408/) (2025)
- [Prediction approaches for partly missing multi-omics covariate data: A literature review and an empirical comparison study](https://mlsanity.com/paper/2302.03991/) (2023)
- [Benchmarking on Tasks That Matter: Dataset Selection for Preserving Model Rankings](https://mlsanity.com/paper/2606.27997/) (2026)
- [Generalizability of experimental studies](https://mlsanity.com/paper/2406.17374/) (2024)
- [Assessing Dutch Syllabification Algorithms and Improving Accuracy by Combining Phonetic and Orthographic Information through Deep Learning](https://mlsanity.com/paper/2605.28834/) (2026)
- [From field-scale to large-scale spectral libraries: Tabular foundation models in soil spectroscopy](https://mlsanity.com/paper/2608.00608/) (2026)

## Abstract (arXiv)

In recent years, the need for neutral benchmark studies that focus on the comparison of methods from computational sciences has been increasingly recognised by the scientific community. While general advice on the design and analysis of neutral benchmark studies can be found in recent literature, certain amounts of flexibility always exist. This includes the choice of data sets and performance measures, the handling of missing performance values and the way the performance values are aggregated over the data sets. As a consequence of this flexibility, researchers may be concerned about how their choices affect the results or, in the worst case, may be tempted to engage in questionable research practices (e.g. the selective reporting of results or the post-hoc modification of design or analysis components) to fit their expectations or hopes. To raise awareness for this issue, we use an example benchmark study to illustrate how variable benchmark results can be when all possible combinations of a range of design and analysis options are considered. We then demonstrate how the impact of each choice on the results can be assessed using multidimensional unfolding. In conclusion, based on previous literature and on our illustrative example, we claim that the multiplicity of design and analysis options combined with questionable research practices lead to biased interpretations of benchmark results and to over-optimistic conclusions. This issue should be considered by computational researchers when designing and analysing their benchmark studies and by the scientific community in general in an effort towards more reliable benchmark results.