mlsanityData to 7 Oct 2026

AI and the Everything in the Whole Wide World Benchmark

arXiv 2111.15366 ↗First posted 26 Nov 2021cs.LG cs.AI cs.PFcited by 340

Not on the map. Why?

Cited by 21 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

There is a tendency across different subfields in AI to valorize a small collection of influential benchmarks. These benchmarks operate as stand-ins for a range of anointed common problems that are frequently framed as foundational milestones on the path towards flexible and generalizable AI systems. State-of-the-art performance on these benchmarks is widely understood as indicative of progress towards these long-term goals. In this position paper, we explore the limits of such benchmarks in order to reveal the construct validity issues in their framing as the functionally "general" broad measures of progress they are set up to be.

Key facts. Cited by 340 papers in the map (data through 7 Oct 2026). Builds on 49 of its 128 references that are in the map. First posted on arXiv 26 Nov 2021.

Builds on

49 of 128 references.

References · 49

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  5. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  6. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  7. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  8. 2018Datasheets for Datasetscited by 1,371
  9. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  10. 2018Model Cards for Model Reportingcited by 1,068
  11. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  12. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
37 more
  1. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 504
  2. 2018Deep Learning for Generic Object Detection: A Surveycited by 392
  3. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 317
  4. 2019Measurement and Fairnesscited by 299
  5. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  6. 2020Data and its (dis)contents: A survey of dataset development and use in machine learning researchcited by 259
  7. 2020Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learningcited by 246
  8. 2020Experience Grounds Languagecited by 241
  9. 2020Large image datasets: A pyrrhic win for computer vision?cited by 237
  10. 2018The Natural Language Decathlon: Multitask Learning as Question Answeringcited by 235
  11. 2019Probing Neural Network Comprehension of Natural Language Argumentscited by 212
  12. 2021What Will it Take to Fix Benchmarking in Natural Language Understanding?cited by 200
  13. 2016What makes ImageNet good for transfer learning?cited by 192
  14. 2020Social Biases in NLP Models as Barriers for Persons with Disabilitiescited by 182
  15. 2019Show Your Work: Improved Reporting of Experimental Resultscited by 164
  16. 2020Characterising Bias in Compressed Modelscited by 162
  17. 2020The Hardware Lotterycited by 160
  18. 2017No Classification without Representation: Assessing Geodiversity Issues in Open Data Sets for the Developing Worldcited by 152
  19. 2018Counterfactual Fairness in Text Classification through Robustnesscited by 150
  20. 2019Does Object Recognition Work for Everyone?cited by 149
  21. 2019Lessons from Archives: Strategies for Collecting Sociocultural Data in Machine Learningcited by 116
  22. 2021Do Datasets Have Politics? Disciplinary Values in Computer Vision Dataset Developmentcited by 85
  23. 2019Diversity in Facescited by 77
  24. 2018The History Began from AlexNet: A Comprehensive Survey on Deep Learning Approachescited by 74
  25. 2019Assessing the Benchmarking Capacity of Machine Reading Comprehension Datasetscited by 46
  26. 2020Targeting the Benchmark: On Methodology in Current Natural Language Processing Researchcited by 44
  27. 2021Why AI is Harder Than We Thinkcited by 40
  28. 2021Disembodied Machine Learning: On the Illusion of Objectivity in NLPcited by 37
  29. 2020Utility is in the Eye of the User: A Critique of NLP Leaderboardscited by 28
  30. 2019Value-laden Disciplinary Shifts in Machine Learningcited by 25
  31. 2017Towards Linguistically Generalizable NLP Systems: A Workshop and Shared Taskcited by 24
  32. 2020A Hierarchy of Limitations in Machine Learningcited by 19
  33. 2020It's not a Non-Issue: Negation as a Source of Error in Machine Translationcited by 19
  34. 2021Underreporting of errors in NLG output, and what to do about itcited by 19
  35. 2020Beyond Leaderboards: A survey of methods for revealing weaknesses in Natural Language Inference data and modelscited by 11
  36. 2020General Purpose Text Embeddings from Pre-trained Language Models for Scalable Inferencecited by 9
  37. 2020TuringAdvice: A Generative and Dynamic Evaluation of Language Usecited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 39; 2023: 64; 2024: 60; 2025: 74; 2026: 102.

340 papers.

Sort:

By area:

  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  2. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  3. 2023PaLM 2 Technical Reportcited by 1,154
  4. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  5. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
  6. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 281
  7. 2023Challenges and Applications of Large Language Modelscited by 251
  8. 2023Can Large Language Models Transform Computational Social Science?cited by 245
  9. 2023Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solvecited by 175
  10. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 172
  11. 2023The Capacity for Moral Self-Correction in Large Language Modelscited by 171
  12. 2024Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Toolscited by 170
38 more
  1. 2023Holistic Evaluation of Text-To-Image Modelscited by 160
  2. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 154
  3. 2023Neural Architecture Search: Insights from 1000 Paperscited by 136
  4. 2023Harms from Increasingly Agentic Algorithmic Systemscited by 134
  5. 2023Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineeringcited by 125
  6. 2023AI Transparency in the Age of LLMs: A Human-Centered Research Roadmapcited by 119
  7. 2024CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmarkcited by 115
  8. 2024BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practicescited by 112
  9. 2025Measuring what Matters: Construct Validity in Large Language Model Benchmarkscited by 102
  10. 2021Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Researchcited by 100
  11. 2023Levels of AGI for Operationalizing Progress on the Path to AGIcited by 94
  12. 2025Establishing Best Practices for Building Rigorous Agentic Benchmarkscited by 94
  13. 2025Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivitycited by 93
  14. 2023Evaluating the Social Impact of Generative AI Systems in Systems and Societycited by 91
  15. 2022DataPerf: Benchmarks for Data-Centric AI Developmentcited by 89
  16. 2022The Fallacy of AI Functionalitycited by 88
  17. 2023A Benchmark for Learning to Translate a New Language from One Grammar Bookcited by 74
  18. 2023LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domaincited by 68
  19. 2022Leakage and the Reproducibility Crisis in ML-based Sciencecited by 63
  20. 2023The Gradient of Generative AI Release: Methods and Considerationscited by 63
  21. 2022Do Large Language Models know what humans know?cited by 60
  22. 2025Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluationcited by 57
  23. 2023Rethinking Model Evaluation as Narrowing the Socio-Technical Gapcited by 56
  24. 2024Auxiliary task demands mask the capabilities of smaller language modelscited by 54
  25. 2022FairLex: A Multilingual Benchmark for Evaluating Fairness in Legal Text Processingcited by 48
  26. 2022Evaluation Gaps in Machine Learning Practicecited by 48
  27. 2024garak: A Framework for Security Probing Large Language Modelscited by 48
  28. 2025Large Language Models Pass the Turing Testcited by 48
  29. 2024A Decade's Battle on Dataset Bias: Are We There Yet?cited by 46
  30. 2024People cannot distinguish GPT-4 from a human in a Turing testcited by 46
  31. 2025Toward an Evaluation Science for Generative AI Systemscited by 42
  32. 2023Data Contamination Through the Lens of Timecited by 39
  33. 2023Does GPT-4 pass the Turing test?cited by 37
  34. 2022GEMv2: Multilingual NLG Benchmarking in a Single Line of Codecited by 36
  35. 2025Medical Large Language Model Benchmarks Should Prioritize Construct Validitycited by 36
  36. 2023Revealing the structure of language model capabilitiescited by 35
  37. 2023Lost in Translation: Large Language Models in Non-English Content Analysiscited by 34
  38. 2024Benchmarks as Microscopes: A Call for Model Metrologycited by 33

50 most cited of 340 All 340 (JSON)