mlsanityData to 7 Oct 2026

Mapping global dynamics of benchmark creation and saturation in artificial intelligence

arXiv 2203.04592 ↗First posted 9 Mar 2022cs.AI cs.CL cs.CVcited by 71

Not on the map. Why?

Abstract (arXiv)

Benchmarks are crucial to measuring and steering progress in artificial intelligence (AI). However, recent studies raised concerns over the state of AI benchmarking, reporting issues such as benchmark overfitting, benchmark saturation and increasing centralization of benchmark dataset creation. To facilitate monitoring of the health of the AI benchmarking ecosystem, we introduce methodologies for creating condensed maps of the global dynamics of benchmark creation and saturation. We curated data for 3765 benchmarks covering the entire domains of computer vision and natural language processing, and show that a large fraction of benchmarks quickly trended towards near-saturation, that many benchmarks fail to find widespread utilization, and that benchmark performance gains for different AI tasks were prone to unforeseen bursts. We analyze attributes associated with benchmark popularity, and conclude that future benchmarks should emphasize versatility, breadth and real-world utility.

Key facts. Cited by 71 papers in the map (data through 7 Oct 2026). Builds on 16 of its 36 references that are in the map. First posted on arXiv 9 Mar 2022.

Builds on

16 of 36 references.

References · 16

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  3. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  4. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  5. 2014Convolutional Neural Networks for Sentence Classificationcited by 1,124
  6. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  7. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  8. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
  9. 2021What Will it Take to Fix Benchmarking in Natural Language Understanding?cited by 200
  10. 2021The Benchmark Lotterycited by 133
  11. 2021Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Researchcited by 100
  12. 2021The AI Index 2021 Annual Reportcited by 34
4 more
  1. 2022Benchmark datasets driving artificial intelligence development fail to capture the needs of medical professionalscited by 16
  2. 2020A critical analysis of metrics used for measuring progress in artificial intelligencecited by 14
  3. 2021A curated, ontology-based, large-scale knowledge graph of artificial intelligence tasks and benchmarkscited by 11
  4. 2019WebProt'eg'e: A Cloud-Based Ontology Editorcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 4; 2024: 14; 2025: 17; 2026: 31.

71 papers.

Sort:

By area:

  1. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  2. 2025Humanity's Last Examcited by 467
  3. 2024BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practicescited by 112
  4. 2025Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluationcited by 57
  5. 2022Melting Pot 2.0cited by 37
  6. 2024Benchmarks as Microscopes: A Call for Model Metrologycited by 33
  7. 2026When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturationcited by 29
  8. 2025Virology Capabilities Test (VCT): A Multimodal Virology Q&A Benchmarkcited by 27
  9. 2024Inherent Trade-Offs between Diversity and Stability in Multi-Task Benchmarkscited by 26
  10. 2022Law Informs Code: A Legal Informatics Approach to Aligning Artificial Intelligence with Humanscited by 23
  11. 2024Improving Model Evaluation using SMART Filtering of Benchmark Datasetscited by 20
  12. 2024A Philosophical Introduction to Language Models - Part II: The Way Forwardcited by 17
38 more
  1. 2025Probing the Critical Point (CritPt) of AI Reasoning: a Frontier Physics Research Benchmarkcited by 16
  2. 2026The Red Queen G"odel Machine: Co-Evolving Agents and Their Evaluatorscited by 16
  3. 2022Vote'n'Rank: Revision of Benchmarking with Social Choice Theorycited by 15
  4. 2024ONEBench to Test Them All: Sample-Level Benchmarking Over Open-Ended Capabilitiescited by 13
  5. 2025Thinking beyond the anthropomorphic paradigm benefits LLM researchcited by 13
  6. 2022TAPE: Assessing Few-shot Russian Language Understandingcited by 11
  7. 2024Statistical Multicriteria Benchmarking via the GSD-Frontcited by 11
  8. 2024Efficient Lifelong Model Evaluation in an Era of Rapid Progresscited by 10
  9. 2024Towards Better Open-Ended Text Generation: A Multicriteria Evaluation Frameworkcited by 9
  10. 2025Statistical Multicriteria Evaluation of LLM-Generated Textcited by 9
  11. 2026No One Knows the State of the Art in Geospatial Foundation Modelscited by 9
  12. 2023Intrinsic Self-Supervision for Data Quality Auditscited by 8
  13. 2024RES-Q: Evaluating Code-Editing Large Language Model Systems at the Repository Scalecited by 7
  14. 2025Pretraining on the Test Set Is No Longer All You Need: A Debate-Driven Approach to QA Benchmarkscited by 7
  15. 2025Benchmarking is Broken -- Don't Let AI be its Own Judgecited by 7
  16. 2026Computational Hermeneutics: Evaluating generative AI as a cultural technologycited by 7
  17. 2024On the Workflows and Smells of Leaderboard Operations (LBOps): An Exploratory Study of Foundation Model Leaderboardscited by 6
  18. 2024Thematic Analysis with Open-Source Generative AI and Machine Learning: A New Method for Inductive Qualitative Codebook Developmentcited by 6
  19. 2026Mediocrity is the key for LLM as a Judge Anchor Selectioncited by 6
  20. 2024Philosophy of Cognitive Science in the Age of Deep Learningcited by 5
  21. 2025Auditing the Ethical Logic of Generative AI Modelscited by 5
  22. 2025A Novel Ophthalmic Benchmark for Evaluating Multimodal Large Language Models with Fundus Photographs and OCT Imagescited by 4
  23. 2025Lessons from the trenches on evaluating machine-learning systems in materials sciencecited by 4
  24. 2025Multimodal Language Models as Text-to-Image Model Evaluatorscited by 3
  25. 2026EnactToM: An Evolving Benchmark for Functional Theory of Mind in Embodied Agentscited by 3
  26. 2026Can We Trust Item Response Theory for AI Evaluation?cited by 3
  27. 2023BenchMD: A Benchmark for Unified Learning on Medical Images and Sensorscited by 2
  28. 2024Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systemscited by 2
  29. 2025Meek Models Shall Inherit the Earthcited by 2
  30. 2025AgentExpt: Automating AI Experiment Design with LLM-based Resource Retrieval Agentcited by 2
  31. 2026Robustness as an Emergent Property of Task Performancecited by 2
  32. 2022ASR in German: A Detailed Error Analysiscited by 1
  33. 2023Operationalising the Definition of General Purpose AI Systems: Assessing Four Approachescited by 1
  34. 2026CLARIN-PT-LDB: An Open LLM Leaderboard for Portuguese to assess Language, Culture and Civilitycited by 1
  35. 2026AI Evaluation Should Require Standardized Item-Level Data Releasescited by 1
  36. 2026Offline Preference-Based Trajectory Evaluationcited by 1
  37. 2025Quantum Ensembling Methods for Healthcare and Life Sciencecited by 0
  38. 2025Deprecating Benchmarks: Criteria and Frameworkcited by 0

50 most cited of 71 All 71 (JSON)