mlsanityData to 7 Oct 2026

Accounting for Variance in Machine Learning Benchmarks

arXiv 2103.03098 ↗First posted 1 Mar 2021cs.LG stat.MLcited by 195

Not on the map. Why?

Abstract (arXiv)

Strong empirical evidence that one machine-learning algorithm A outperforms another one B ideally calls for multiple trials optimizing the learning pipeline over sources of variation such as data sampling, data augmentation, parameter initialization, and hyperparameters choices. This is prohibitively expensive, and corners are cut to reach conclusions. We model the whole benchmarking process, revealing that variance due to data sampling, parameter initialization and hyperparameter choice impact markedly the results. We analyze the predominant comparison methods used today in the light of this variance. We show a counter-intuitive result that adding more sources of variation to an imperfect estimator approaches better the ideal estimator at a 51 times reduction in compute cost. Building on these results, we study the error rate of detecting improvements, on five different deep-learning tasks/architectures. This study leads us to propose recommendations for performance comparisons.

Key facts. Cited by 195 papers in the map (data through 7 Oct 2026). Builds on 24 of its 76 references that are in the map. First posted on arXiv 1 Mar 2021.

Builds on

24 of 76 references.

References · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  6. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  7. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  8. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  9. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  10. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
  11. 2017Deep Reinforcement Learning that Matterscited by 727
  12. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
12 more
  1. 2020What is the State of Neural Network Pruning?cited by 578
  2. 2017Progressive Neural Architecture Searchcited by 483
  3. 2020A Metric Learning Reality Checkcited by 317
  4. 2017Are GANs Created Equal? A Large-Scale Studycited by 267
  5. 2019MLPerf Inference Benchmarkcited by 195
  6. 2019Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approachescited by 181
  7. 2017On the State of the Art of Evaluation in Neural Language Modelscited by 103
  8. 2019A Step Toward Quantifying Independently Reproducible Machine Learning Researchcited by 77
  9. 2017Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Taggingcited by 71
  10. 2017Knowledge Base Completion: Baselines Strike Backcited by 39
  11. 2017Replicability Analysis for Natural Language Processing: Testing Significance with Multiple Datasetscited by 32
  12. 2020Research Reproducibility as a Survival Analysiscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 18; 2022: 20; 2023: 25; 2024: 26; 2025: 20; 2026: 86.

195 papers.

Sort:

By area:

  1. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  2. 2022Why do tree-based models still outperform deep learning on tabular data?cited by 198
  3. 2021The Benchmark Lotterycited by 133
  4. 2023Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineeringcited by 125
  5. 2021HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPOcited by 111
  6. 2021Torch.manual_seed(3407) is all you need: On the influence of random seeds in deep learning architectures for computer visioncited by 76
  7. 2021DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularizationcited by 66
  8. 2023Elo Uncovered: Robustness and Best Practices in Language Model Evaluationcited by 58
  9. 2021Randomness In Neural Network Training: Characterizing The Impact of Toolingcited by 57
  10. 2023GADBench: Revisiting and Benchmarking Supervised Graph Anomaly Detectioncited by 57
  11. 2024Good Seed Makes a Good Crop: Discovering Secret Seeds in Text-to-Image Diffusion Modelscited by 47
  12. 2021How I failed machine learning in medical imaging -- shortcomings and recommendationscited by 37
38 more
  1. 2022deep-significance - Easy and Meaningful Statistical Significance Testing in the Age of Neural Networkscited by 28
  2. 2022Sources of Irreproducibility in Machine Learning: A Reviewcited by 28
  3. 2021Learning with Holographic Reduced Representationscited by 25
  4. 2021Relating the Partial Dependence Plot and Permutation Feature Importance to the Data Generating Processcited by 24
  5. 2023Improving Convergence and Generalization Using Parameter Symmetriescited by 23
  6. 2023On the Variance of Neural Network Training with respect to Test Sets and Distributionscited by 20
  7. 2023Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Modelcited by 19
  8. 2024Position: Why We Must Rethink Empirical Research in Machine Learningcited by 19
  9. 2021Learning Pessimism for Robust and Efficient Off-Policy Reinforcement Learningcited by 18
  10. 2025The Rise of Small Language Models in Healthcare: A Comprehensive Surveycited by 18
  11. 2022The worst of both worlds: A comparative analysis of errors in learning from data in psychology and machine learningcited by 17
  12. 2023The Dataset Multiplicity Problem: How Unreliable Data Impacts Predictionscited by 17
  13. 2025Systemizing Multiplicity: The Curious Case of Arbitrariness in Machine Learningcited by 17
  14. 2021A Framework for Cluster and Classifier Evaluation in the Absence of Reference Labelscited by 16
  15. 2023Can Fairness be Automated? Guidelines and Opportunities for Fairness-aware AutoMLcited by 16
  16. 2024Reinforcing Language Agents via Policy Optimization with Action Decompositioncited by 16
  17. 2024Public Domain 12M: A Highly Aesthetic Image-Text Dataset with Novel Governance Mechanismscited by 15
  18. 2023A Survey on Stability of Learning with Limited Labelled Data and its Sensitivity to the Effects of Randomnesscited by 14
  19. 2026SafetyRepro: Configuration-Conditional Rank Instability on Alignment Benchmarkscited by 14
  20. 2023REFORMS: Reporting Standards for Machine Learning Based Sciencecited by 13
  21. 2024Entropy-Regularized Token-Level Policy Optimization for Language Agent Reinforcementcited by 12
  22. 2024Confidence intervals uncovered: Are we ready for real-world medical imaging AI?cited by 12
  23. 2023Reproducibility in Multiple Instance Learning: A Case For Algorithmic Unit Testscited by 11
  24. 2022A Revealing Large-Scale Evaluation of Unsupervised Anomaly Detection Algorithmscited by 10
  25. 2024Evaluation of pseudo-healthy image reconstruction for anomaly detection with deep generative models: Application to brain FDG PETcited by 10
  26. 2024Benchmark Data Repositories for Better Benchmarkingcited by 10
  27. 2025Statistical Uncertainty Quantification for Aggregate Performance Metrics in Machine Learning Benchmarkscited by 10
  28. 2022Experimental Standards for Deep Learning in Natural Language Processing Researchcited by 9
  29. 2026No One Knows the State of the Art in Geospatial Foundation Modelscited by 9
  30. 2024What Do Machine Learning Researchers Mean by "Reproducible"?cited by 8
  31. 2022Does the Market of Citations Reward Reproducible Work?cited by 7
  32. 2022A Siren Song of Open Source Reproducibilitycited by 7
  33. 2024Reshuffling Resampling Splits Can Improve Generalization of Hyperparameter Optimizationcited by 7
  34. 2025When +1% Is Not Enough: A Paired Bootstrap Protocol for Evaluating Small Improvementscited by 7
  35. 2021When are Deep Networks really better than Decision Forests at small sample sizes, and how?cited by 6
  36. 2021Empirical Evaluation of Deep Learning Models for Knowledge Tracing: Of Hyperparameters and Metrics on Performance and Replicabilitycited by 6
  37. 2023Model Reporting for Certifiable AI: A Proposal from Merging EU Regulation into AI Developmentcited by 6
  38. 2024Is Function Similarity Over-Engineered? Building a Benchmarkcited by 6

50 most cited of 195 All 195 (JSON)