mlsanityData to 7 Oct 2026

When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation

arXiv 2602.16763 ↗First posted 18 Feb 2026cs.AIcited by 29

Not on the map. Why?

Abstract (arXiv)

Artificial intelligence benchmarks are an important mechanism to measure model progress and guide deployment decisions. However, benchmarks quickly "saturate", making it difficult to differentiate models and diminishing their long-term value. In this study, we define benchmark saturation and analyze it across 60 language model benchmarks using 14 properties that relate to saturation. We find that nearly half of our benchmarks exhibit saturation, with rates increasing with age. Further, we find that resilience to saturation is impacted by expert-curation, not by public test data. Our results suggest that design choices can extend benchmark longevity and inform more durable evaluation approaches.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 78 of its 107 references that are in the map. First posted on arXiv 18 Feb 2026.

Builds on

78 of 107 references.

References · 78

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  10. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  11. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  12. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
38 more
  1. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  2. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  3. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  4. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  5. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  6. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  7. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  8. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  9. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  10. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  11. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  12. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  13. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  14. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  15. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  16. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  17. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  18. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  19. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  20. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  21. 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 558
  22. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
  23. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  24. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
  25. 2018QuAC : Question Answering in Contextcited by 379
  26. 2024Measuring short-form factuality in large language modelscited by 345
  27. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 340
  28. 2023LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Modelscited by 328
  29. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 300
  30. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 242
  31. 2024FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AIcited by 227
  32. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  33. 2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 204
  34. 2024Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluationcited by 181
  35. 2024Are We Done with MMLU?cited by 178
  36. 2025ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systemscited by 171
  37. 2023SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialectscited by 149
  38. 2023Investigating Data Contamination in Modern Benchmarks for Large Language Modelscited by 136

50 most cited of 78 All 78 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 29.

29 papers.

Sort:

By area:

  1. 2026Linear Probes Detect Task Format, Not Reasoning Mode in Language Model Hidden Statescited by 7
  2. 2026EnactToM: An Evolving Benchmark for Functional Theory of Mind in Embodied Agentscited by 3
  3. 2026MMGist: A Comprehensive Multimodal Benchmark for 2027cited by 3
  4. 2026Life After Benchmark Saturation: A Case Study of CORE-Benchcited by 3
  5. 2026CivBench: Progress-Based Evaluation for LLMs' Strategic Decision-Making in Civilization Vcited by 2
  6. 2026Every Eval Ever: A Unifying Schema and Community Repository for AI Evaluation Resultscited by 2
  7. 2026AI Evaluation Should Require Standardized Item-Level Data Releasescited by 1
  8. 2026The GaoYao Benchmark: A Comprehensive Framework for Evaluating Multilingual and Multicultural Abilities of Large Language Modelscited by 1
  9. 2026Offline Preference-Based Trajectory Evaluationcited by 1
  10. 2026Augmenting Text to Increase Translation Difficultycited by 1
  11. 2026Last Translation Benchmarkcited by 1
  12. 2026Benchmarked Yet Not Measured -- Generative AI Should be Evaluated Against Real-World Utilitycited by 0
17 more
  1. 2026The Growing Pains of Frontier Models: When Leaderboards Stop Separating and What to Measure Nextcited by 0
  2. 2026SEAL: Can Saturated Benchmarks Be Revived by LLM-as-a-Meta-Judge?cited by 0
  3. 2026SAKE: Software Architectural Knowledge Evaluation Benchmark for Large Language Modelscited by 0
  4. 2026Limits of Reliability and Scaling in Language Modelscited by 0
  5. 2026Rethinking Transfer in Continual Learning: A Replay-Based Realisationcited by 0
  6. 2026Economic Evaluations of Language Modelscited by 0
  7. 2026InfoOps Bench: A live information operations safety benchmarkcited by 0
  8. 2026Multi-Dimensional Assessment for AI Cognition (MAAC): A Theoretical Framework for Process-Oriented Cognitive Evaluation of Text-Based AI Systemscited by 0
  9. 2026Frontier AI Forecasting Has a Measurement Problem: An Audit of Progress Evidencecited by 0
  10. 2026One Capability or Many? Structural and Predictive Tests of Benchmark Validity Disagree About Economic Benchmarks for Frontier AIcited by 0
  11. 2026Auditing MCQA Benchmarks through Probability Landscapescited by 0
  12. 2026Benchmark Radar: A Living Database and Search Engine for AI Benchmarks and Evaluationcited by 0
  13. 2026Do Not Trust the Benchmark: Limitations of General LLM Rankings and a Case for Task-Specific Evaluationcited by 0
  14. 2026UltraBench 2: Towards Robust Evaluation of Vision Foundation Models on Ultrasoundcited by 0
  15. 2026RECLAIM: Can Agents Reproduce the Claims of Machine Learning Papers?cited by 0
  16. 2026Beneath the Scores: Rethinking Hallucination Evaluation for Video Understanding Modelscited by 0
  17. 2026What Makes Something Hard(er)? Explaining Question Difficulty in Natural Languagecited by 0