mlsanityData to 7 Oct 2026

The Benchmark Lottery

arXiv 2107.07002 ↗First posted 14 Jul 2021cs.LG cs.AI cs.CLcited by 133

Not on the map. Why?

Abstract (arXiv)

The world of empirical machine learning (ML) strongly relies on benchmarks in order to determine the relative effectiveness of different algorithms and methods. This paper proposes the notion of "a benchmark lottery" that describes the overall fragility of the ML benchmarking process. The benchmark lottery postulates that many factors, other than fundamental algorithmic superiority, may lead to a method being perceived as superior. On multiple benchmark setups that are prevalent in the ML community, we show that the relative performance of algorithms may be altered significantly simply by choosing different benchmark tasks, highlighting the fragility of the current paradigms and potential fallacious interpretation derived from benchmarking ML methods. Given that every benchmark makes a statement about what it perceives to be important, we argue that this might lead to biased progress in the community. We discuss the implications of the observed phenomena and provide recommendations on mitigating them using multiple machine learning domains and communities as use cases, including natural language processing, computer vision, information retrieval, recommender systems, and reinforcement learning.

Key facts. Cited by 133 papers in the map (data through 7 Oct 2026). Builds on 91 of its 150 references that are in the map. First posted on arXiv 14 Jul 2021.

Builds on

91 of 150 references.

References · 91

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  10. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  11. 2013Auto-Encoding Variational Bayescited by 3,980
  12. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
38 more
  1. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  3. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  4. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  5. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  6. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  8. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  9. 2013Describing Textures in the Wildcited by 2,110
  10. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  11. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  12. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  13. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  14. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  15. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  16. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  17. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  18. 2019Natural Adversarial Examplescited by 1,401
  19. 2017Neural Collaborative Filteringcited by 1,108
  20. 2018DeepMind Control Suitecited by 935
  21. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
  22. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  23. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  24. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  25. 2017Deep Reinforcement Learning that Matterscited by 727
  26. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  27. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  28. 2018Universal Transformerscited by 613
  29. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  30. 2016Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filteringcited by 517
  31. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
  32. 2018Rotation Equivariant CNNs for Digital Pathologycited by 420
  33. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 419
  34. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  35. 2021Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarkscited by 388
  36. 2017Wasserstein Auto-Encoderscited by 371
  37. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 350
  38. 2020Are we done with ImageNet?cited by 329

50 most cited of 91 All 91 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 18; 2023: 18; 2024: 16; 2025: 17; 2026: 55.

133 papers.

Sort:

By area:

  1. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  2. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  3. 2022UL2: Unifying Language Learning Paradigmscited by 292
  4. 2023Challenges and Applications of Large Language Modelscited by 251
  5. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 172
  6. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 154
  7. 2023Harms from Increasingly Agentic Algorithmic Systemscited by 134
  8. 2024When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboardscited by 134
  9. 2021Scale Efficiently: Insights from Pre-training and Fine-tuning Transformerscited by 105
  10. 2021Exploring the Limits of Large Scale Pre-trainingcited by 104
  11. 2022Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?cited by 103
  12. 2021The Efficiency Misnomercited by 98
38 more
  1. 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibilitycited by 89
  2. 2024Better than classical? The subtle art of benchmarking quantum machine learning modelscited by 82
  3. 2023Unleashing the potential of prompt engineering for large language modelscited by 72
  4. 2022Mapping global dynamics of benchmark creation and saturation in artificial intelligencecited by 71
  5. 2021YAHPO Gym -- An Efficient Multi-Objective Multi-Fidelity Benchmark for Hyperparameter Optimizationcited by 61
  6. 2025Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluationcited by 57
  7. 2021SCENIC: A JAX Library for Computer Vision Research and Beyondcited by 56
  8. 2023Active Learning Principles for In-Context Learning with Large Language Modelscited by 53
  9. 2024Quantifying Variance in Evaluation Benchmarkscited by 52
  10. 2021The Dangers of Underclaiming: Reasons for Caution When Reporting How NLP Systems Failcited by 38
  11. 2022GEMv2: Multilingual NLG Benchmarking in a Single Line of Codecited by 36
  12. 2024Vibe-Eval: A hard evaluation suite for measuring progress of multimodal language modelscited by 35
  13. 2026AIRS-Bench: a Suite of Tasks for Frontier AI Research Science Agentscited by 32
  14. 2022deep-significance - Easy and Meaningful Statistical Significance Testing in the Age of Neural Networkscited by 28
  15. 2025RefactorBench: Evaluating Stateful Reasoning in Language Agents Through Codecited by 28
  16. 2024Inherent Trade-Offs between Diversity and Stability in Multi-Task Benchmarkscited by 26
  17. 2023It Takes Two to Tango: Navigating Conceptualizations of NLP Tasks and Measurements of Performancecited by 25
  18. 2023Adaptive Computation with Elastic Input Sequencecited by 22
  19. 2022The Role of Machine Learning in Cybersecuritycited by 21
  20. 2021A Tale Of Two Long Tailscited by 19
  21. 2024Position: Why We Must Rethink Empirical Research in Machine Learningcited by 19
  22. 2022The worst of both worlds: A comparative analysis of errors in learning from data in psychology and machine learningcited by 17
  23. 2023Can Fairness be Automated? Guidelines and Opportunities for Fairness-aware AutoMLcited by 16
  24. 2022SoK: The Impact of Unlabelled Data in Cyberthreat Detectioncited by 15
  25. 2022Vote'n'Rank: Revision of Benchmarking with Social Choice Theorycited by 15
  26. 2022Multi-CLS BERT: An Efficient Alternative to Traditional Ensemblingcited by 14
  27. 2023Towards More Robust NLP System Evaluation: Handling Missing Scores in Benchmarkscited by 14
  28. 2023A Survey on Stability of Learning with Limited Labelled Data and its Sensitivity to the Effects of Randomnesscited by 14
  29. 2024ONEBench to Test Them All: Sample-Level Benchmarking Over Open-Ended Capabilitiescited by 13
  30. 2026Do Androids Dream of Breaking the Game? Systematically Auditing AI Agent Benchmarks with BenchJackcited by 13
  31. 2023Toward a Theory of Causation for Interpreting Neural Code Modelscited by 12
  32. 2025The Benchmarking Epistemology: Validity Theory for Evaluating Machine Learning Modelscited by 12
  33. 2022Towards Better User Studies in Computer Graphics and Visioncited by 11
  34. 2025Is GPT-OSS Good? A Comprehensive Evaluation of OpenAI's Latest Open Source Modelscited by 11
  35. 2022Making Intelligence: Ethical Values in IQ and ML Benchmarkscited by 10
  36. 2024Benchmark Data Repositories for Better Benchmarkingcited by 10
  37. 2025Statistical Uncertainty Quantification for Aggregate Performance Metrics in Machine Learning Benchmarkscited by 10
  38. 2026RewardHackingAgents: Benchmarking Evaluation Integrity for LLM ML-Engineering Agentscited by 10

50 most cited of 133 All 133 (JSON)