# On the role of benchmarking data sets and simulations in method comparison studies

arXiv: https://arxiv.org/abs/2208.01457 · HTML: https://mlsanity.com/paper/2208.01457/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 103 references that are in the map.
- First posted on arXiv 2 Aug 2022.

## Builds on (most cited first, 18 of 18)

- [Benchmarking Graph Neural Networks](https://mlsanity.com/paper/2003.00982/) (2020)
- [AI and the Everything in the Whole Wide World Benchmark](https://mlsanity.com/paper/2111.15366/) (2021)
- [Deep Learning for ECG Analysis: Benchmarks and Insights from PTB-XL](https://mlsanity.com/paper/2004.13701/) (2020)
- [Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Research](https://mlsanity.com/paper/2112.01716/) (2021)
- [Leakage and the Reproducibility Crisis in ML-based Science](https://mlsanity.com/paper/2207.07048/) (2022)
- [It's COMPASlicated: The Messy Relationship between RAI Datasets and Algorithmic Fairness Benchmarks](https://mlsanity.com/paper/2106.05498/) (2021)
- [Using simulation studies to evaluate statistical methods](https://mlsanity.com/paper/1712.03198/) (2017)
- [Over-optimism in benchmark studies and the multiplicity of design and analysis options when interpreting their results](https://mlsanity.com/paper/2106.02447/) (2021)
- [OpenML: An R Package to Connect to the Machine Learning Platform OpenML](https://mlsanity.com/paper/1701.01293/) (2017)
- [Validation of cluster analysis results on validation data: A systematic framework](https://mlsanity.com/paper/2103.01281/) (2021)
- [Stability selection for component-wise gradient boosting in multiple dimensions](https://mlsanity.com/paper/1611.10171/) (2016)
- [Ultrasound segmentation using U-Net: learning from simulated data and testing on real data](https://mlsanity.com/paper/1904.11031/) (2019)
- [Is there a role for statistics in artificial intelligence?](https://mlsanity.com/paper/2009.09070/) (2020)
- [Causal inference methods for small non-randomized studies: Methods and an application in COVID-19](https://mlsanity.com/paper/2007.15991/) (2020)
- [Pitfalls and potentials in simulation studies: Questionable research practices in comparative simulation studies allow for spurious claims of superiority of any method](https://mlsanity.com/paper/2203.13076/) (2022)
- [The Wild Bootstrap for Multivariate Nelson-Aalen Estimators](https://mlsanity.com/paper/1602.02071/) (2016)
- [Likelihood-based meta-analysis with few studies: Empirical and simulation studies](https://mlsanity.com/paper/1807.09037/) (2018)
- [Let's practice what we preach: Planning and interpreting simulation studies with design and analysis of experiments](https://mlsanity.com/paper/2111.13737/) (2021)

## Built on by (most cited first, 7 of 7)

- [Comparing statistical and machine learning methods for time series forecasting in data-driven logistics -- A simulation study](https://mlsanity.com/paper/2303.07139/) (2023)
- [Evaluating tree-based imputation methods as an alternative to MICE PMM for drawing inference in empirical studies](https://mlsanity.com/paper/2401.09602/) (2024)
- [Which Imputation Fits Which Feature Selection Method? A Survey-Based Simulation Study](https://mlsanity.com/paper/2412.13570/) (2024)
- [Frugal, Flexible, Faithful: Causal Data Simulation via Frengression](https://mlsanity.com/paper/2508.01018/) (2025)
- [Controllable Generative Sandbox for Causal Inference](https://mlsanity.com/paper/2603.03587/) (2026)
- [Assessing Dutch Syllabification Algorithms and Improving Accuracy by Combining Phonetic and Orthographic Information through Deep Learning](https://mlsanity.com/paper/2605.28834/) (2026)
- [Beyond Point Prediction: Artificial Representative Trees with Uncertainty](https://mlsanity.com/paper/2609.24528/) (2026)

## Abstract (arXiv)

Method comparisons are essential to provide recommendations and guidance for applied researchers, who often have to choose from a plethora of available approaches. While many comparisons exist in the literature, these are often not neutral but favour a novel method. Apart from the choice of design and a proper reporting of the findings, there are different approaches concerning the underlying data for such method comparison studies. Most manuscripts on statistical methodology rely on simulation studies and provide a single real-world data set as an example to motivate and illustrate the methodology investigated. In the context of supervised learning, in contrast, methods are often evaluated using so-called benchmarking data sets, i.e. real-world data that serve as gold standard in the community. Simulation studies, on the other hand, are much less common in this context. The aim of this paper is to investigate differences and similarities between these approaches, to discuss their advantages and disadvantages and ultimately to develop new approaches to the evaluation of methods picking the best of both worlds. To this aim, we borrow ideas from different contexts such as mixed methods research and Clinical Scenario Evaluation.