mlsanityData to 14 Oct 2026

Position: Why We Must Rethink Empirical Research in Machine Learning

arXiv 2405.02200 ↗First posted 3 May 2024cs.LG stat.MLcited by 19

Not on the map. Why?

Abstract (arXiv)

We warn against a common but incomplete understanding of empirical research in machine learning that leads to non-replicable results, makes findings unreliable, and threatens to undermine progress in the field. To overcome this alarming situation, we call for more awareness of the plurality of ways of gaining knowledge experimentally but also of some epistemic limitations. In particular, we argue most current empirical machine learning research is fashioned as confirmatory research while it should rather be considered exploratory.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 43 of its 155 references that are in the map. First posted on arXiv 3 May 2024.

Builds on

43 of 155 references.

References · 43

  1. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  2. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  3. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  4. 2017Deep Reinforcement Learning that Matterscited by 754
  5. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
  6. 2014OpenML: networked science in machine learningcited by 526
  7. 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 496
  8. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 349
  9. 2020Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)cited by 322
  10. 2017Are GANs Created Equal? A Large-Scale Studycited by 267
  11. 2023When Do Neural Nets Outperform Boosted Trees on Tabular Data?cited by 223
  12. 2021Accounting for Variance in Machine Learning Benchmarkscited by 201
31 more
  1. 2017OpenML Benchmarking Suitescited by 178
  2. 2018Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Samplingcited by 169
  3. 2021The Benchmark Lotterycited by 136
  4. 2021HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPOcited by 112
  5. 2018Troubling Trends in Machine Learning Scholarshipcited by 108
  6. 2017On the State of the Art of Evaluation in Neural Language Modelscited by 104
  7. 2021Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Researchcited by 102
  8. 2021Do Transformer Modifications Transfer Across Implementations and Applications?cited by 87
  9. 2019A Step Toward Quantifying Independently Reproducible Machine Learning Researchcited by 77
  10. 2021Scientific Credibility of Machine Translation Research: A Meta-Evaluation of 769 Paperscited by 71
  11. 2019A Troubling Analysis of Reproducibility and Progress in Recommender Systems Researchcited by 65
  12. 2021Cross-validation: what does it estimate and how well does it do it?cited by 65
  13. 2023Benchmarking Neural Network Training Algorithmscited by 52
  14. 2022NAS-Bench-Suite: NAS Evaluation is (Now) Surprisingly Easycited by 36
  15. 2021How to avoid machine learning pitfalls: a guide for academic researcherscited by 35
  16. 2022The harm of class imbalance corrections for risk prediction models: illustration and simulation using logistic regressioncited by 28
  17. 2020The Fundamental Principles of Reproducibilitycited by 17
  18. 2022The worst of both worlds: A comparative analysis of errors in learning from data in psychology and machine learningcited by 17
  19. 2021Over-optimism in benchmark studies and the multiplicity of design and analysis options when interpreting their resultscited by 15
  20. 2020Large-scale benchmark study of survival prediction methods using multi-omics datacited by 14
  21. 2022To SMOTE, or not to SMOTE?cited by 14
  22. 2019HARK Side of Deep Learning -- From Grad Student Descent to Automated Machine Learningcited by 13
  23. 2018Terminologies for Reproducible Researchcited by 11
  24. 2022A Siren Song of Open Source Reproducibilitycited by 8
  25. 2021Avoiding C-hacking when evaluating survival distribution predictions with discrimination measurescited by 7
  26. 2022Enhancing cluster analysis via topological manifold learningcited by 6
  27. 2022Pitfalls and potentials in simulation studies: Questionable research practices in comparative simulation studies allow for spurious claims of superiority of any methodcited by 2
  28. 2022A geometric framework for outlier detection in high-dimensional datacited by 1
  29. 2022Explaining the optimistic performance evaluation of newly proposed methods: a cross-design validation experimentcited by 1
  30. 2022Phases of methodological research in biostatistics - building the evidence base for new methodscited by 1
  31. 2023Connecting Simple and Precise P-values to Complex and Ambiguous Realitiescited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 13; 2026: 6.

19 papers.

Sort:

By area:

  1. 2025Position: Graph Learning Will Lose Relevance Due To Poor Benchmarkscited by 53
  2. 2025Stop Overvaluing Multi-Agent Debate -- We Must Rethink Evaluation and Embrace Model Heterogeneitycited by 44
  3. 2025Understanding High-Dimensional Bayesian Optimizationcited by 20
  4. 2025STREAM (ChemBio): A Standard for Transparently Reporting Evaluations in AI Model Reportscited by 10
  5. 2025There are no Champions in Supervised Long-Term Time Series Forecastingcited by 8
  6. 2025Rigor in AI: Doing Rigorous AI Work Requires a Broader, Responsible AI-Informed Conception of Rigorcited by 7
  7. 2025Overtuning in Hyperparameter Optimizationcited by 5
  8. 2025HyperSHAP: Shapley Values and Interactions for Explaining Hyperparameter Optimizationcited by 4
  9. 2025The Formalism-Implementation Gap in Reinforcement Learning Researchcited by 3
  10. 2025AI for Just Work: Constructing Diverse Imaginations of AI beyond "Replacing Humans"cited by 2
  11. 2025Reproducibility of Machine Learning-Based Fault Detection and Diagnosis for HVAC Systems in Buildings: An Empirical Studycited by 0
  12. 2025A Function-Centric Perspective on Flat and Sharp Minimacited by 0
7 more
  1. 2025Best Practices For Empirical Meta-Algorithmic Research: Guidelines from the COSEAL Research Networkcited by 0
  2. 2026Position: Evaluation of ECG Representations Must Be Fixedcited by 0
  3. 2026Validity Threats for Foundation Model Researchcited by 0
  4. 2026How Complexity Contributes to Learning Opacity in Machine Learningcited by 0
  5. 2026The Role of Rigor in Artificial Intelligencecited by 0
  6. 2026Evaluating Tiny Recursive Models Across Training for Code Generationcited by 0
  7. 2026Bayesian Optimization with Fisher Information Geometry: Gradient Bounds and Trust-Region Methodscited by 0