# Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluation

arXiv: https://arxiv.org/abs/2502.06559 · HTML: https://mlsanity.com/paper/2502.06559/

## Key facts

- Cited by 60 papers in the map (data through 14 Oct 2026).
- Builds on 71 of its 139 references that are in the map.
- First posted on arXiv 10 Feb 2025.

## Builds on (most cited first, 30 of 71)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://mlsanity.com/paper/2310.03693/) (2023)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [Scalable Extraction of Training Data from (Production) Language Models](https://mlsanity.com/paper/2311.17035/) (2023)
- [Alignment faking in large language models](https://mlsanity.com/paper/2412.14093/) (2024)
- [AI and the Everything in the Whole Wide World Benchmark](https://mlsanity.com/paper/2111.15366/) (2021)
- [State of What Art? A Call for Multi-Prompt LLM Evaluation](https://mlsanity.com/paper/2401.00595/) (2023)
- [Frontier Models are Capable of In-context Scheming](https://mlsanity.com/paper/2412.04984/) (2024)
- [Data and its (dis)contents: A survey of dataset development and use in machine learning research](https://mlsanity.com/paper/2012.05345/) (2020)
- [Data Contamination: From Memorization to Exploitation](https://mlsanity.com/paper/2203.08242/) (2022)
- [Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models](https://mlsanity.com/paper/2205.10770/) (2022)
- [AI Agents That Matter](https://mlsanity.com/paper/2407.01502/) (2024)
- [Hidden Stratification Causes Clinically Meaningful Failures in Machine Learning for Medical Imaging](https://mlsanity.com/paper/1909.12475/) (2019)
- [What Will it Take to Fix Benchmarking in Natural Language Understanding?](https://mlsanity.com/paper/2104.02145/) (2021)
- [Rethinking Benchmark and Contamination for Language Models with Rephrased Samples](https://mlsanity.com/paper/2311.04850/) (2023)
- [Are We Done with MMLU?](https://mlsanity.com/paper/2406.04127/) (2024)
- [Lessons from the Trenches on Reproducible Evaluation of Language Models](https://mlsanity.com/paper/2405.14782/) (2024)
- [Sociotechnical Safety Evaluation of Generative AI Systems](https://mlsanity.com/paper/2310.11986/) (2023)
- [Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text](https://mlsanity.com/paper/2202.06935/) (2022)
- [Benchmark Data Contamination of Large Language Models: A Survey](https://mlsanity.com/paper/2406.04244/) (2024)
- [AI Sandbagging: Language Models can Strategically Underperform on Evaluations](https://mlsanity.com/paper/2406.07358/) (2024)
- [Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasets](https://mlsanity.com/paper/2008.02637/) (2020)
- [When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards](https://mlsanity.com/paper/2402.01781/) (2024)
- [The Benchmark Lottery](https://mlsanity.com/paper/2107.07002/) (2021)
- [Black-Box Access is Insufficient for Rigorous AI Audits](https://mlsanity.com/paper/2401.14446/) (2024)
- [MLPerf Training Benchmark](https://mlsanity.com/paper/1910.01500/) (2019)

## Built on by (most cited first, 30 of 60)

- [Establishing Best Practices for Building Rigorous Agentic Benchmarks](https://mlsanity.com/paper/2507.02825/) (2025)
- [The Singapore Consensus on Global AI Safety Research Priorities](https://mlsanity.com/paper/2506.20702/) (2025)
- [There are no Champions in Supervised Long-Term Time Series Forecasting](https://mlsanity.com/paper/2502.14045/) (2025)
- [Benchmarking is Broken -- Don't Let AI be its Own Judge](https://mlsanity.com/paper/2510.07575/) (2025)
- [Computational Hermeneutics: Evaluating generative AI as a cultural technology](https://mlsanity.com/paper/2604.16403/) (2026)
- [Agentic AI Scientists Are Not Built For Autonomous Scientific Discovery](https://mlsanity.com/paper/2605.08956/) (2026)
- [From Guidelines to Practice: A New Paradigm for Arabic Language Model Evaluation](https://mlsanity.com/paper/2506.01920/) (2025)
- [Benchmark Profiling: Mechanistic Diagnosis of LLM Benchmarks](https://mlsanity.com/paper/2510.01232/) (2025)
- [AIReg-Bench: Benchmarking Language Models That Assess AI Regulation Compliance](https://mlsanity.com/paper/2510.01474/) (2025)
- [ProcCtrlBench: Evaluating Process-Level Defects and Control Preservation in LLM Coding Agents](https://mlsanity.com/paper/2605.20251/) (2026)
- [Measuring Data Science Automation: A Survey of Evaluation Tools for AI Assistants and Agents](https://mlsanity.com/paper/2506.08800/) (2025)
- [The Artificial Intelligence Cognitive Examination: A Survey on the Evolution of Multimodal Evaluation from Recognition to Reasoning](https://mlsanity.com/paper/2510.04141/) (2025)
- [VERA-MH Concept Paper](https://mlsanity.com/paper/2510.15297/) (2025)
- [AI Chatbot Suicide Risk Detection and Response: Human Validation Study of the Open-Source VERA-MH Safety Evaluation](https://mlsanity.com/paper/2602.05088/) (2026)
- [Scalable and Reliable Evaluation of AI Knowledge Retrieval Systems: RIKER and the Coherent Simulated Universe](https://mlsanity.com/paper/2601.08847/) (2025)
- [Agent Benchmarks Fail Public Sector Requirements](https://mlsanity.com/paper/2601.20617/) (2026)
- [AI Evaluation Should Require Standardized Item-Level Data Releases](https://mlsanity.com/paper/2604.03244/) (2026)
- [ComplexConstraints and Beyond: Expert Rubrics for RLVR](https://mlsanity.com/paper/2606.09118/) (2026)
- [Navigating Semantic Relations: Challenges for Language Models in Abstract Common-Sense Reasoning](https://mlsanity.com/paper/2502.14086/) (2025)
- [VLM@school -- Evaluation of AI image understanding on German middle school knowledge](https://mlsanity.com/paper/2506.11604/) (2025)
- [What Does 'Human-Centred AI' Mean?](https://mlsanity.com/paper/2507.19960/) (2025)
- [Position: AI Evaluations Should be Grounded on a Theory of Capability](https://mlsanity.com/paper/2509.19590/) (2025)
- [MoralityGym: A Benchmark for Evaluating Hierarchical Moral Alignment in Sequential Decision-Making Agents](https://mlsanity.com/paper/2602.13372/) (2026)
- [Unexplored flaws in multiple-choice VQA make benchmarking unreliable](https://mlsanity.com/paper/2511.22341/) (2025)
- [What is Safety? Corporate Discourse, Power, and the Politics of Generative AI Safety](https://mlsanity.com/paper/2602.06981/) (2026)
- [From Human-Level AI Tales to AI Leveling Human Scales](https://mlsanity.com/paper/2602.18911/) (2026)
- [BenchBrowser: Retrieving Evidence for Evaluating Benchmark Validity](https://mlsanity.com/paper/2603.18019/) (2026)
- [To Build or Not to Build? Factors that Lead to Non-Development or Abandonment of AI Systems](https://mlsanity.com/paper/2604.28053/) (2026)
- [AI-Assisted Systematization for Evaluating GenAI Systems](https://mlsanity.com/paper/2605.26001/) (2026)
- [What Current AI Benchmarks Leave Unmeasured: Modality, Search, Citations, and Implications (for Safety Evaluations)](https://mlsanity.com/paper/2608.06202/) (2026)

## Abstract (arXiv)

Quantitative Artificial Intelligence (AI) Benchmarks have emerged as fundamental tools for evaluating the performance, capability, and safety of AI models and systems. Currently, they shape the direction of AI development and are playing an increasingly prominent role in regulatory frameworks. As their influence grows, however, so too does concerns about how and with what effects they evaluate highly sensitive topics such as capabilities, including high-impact capabilities, safety and systemic risks. This paper presents an interdisciplinary meta-review of about 100 studies that discuss shortcomings in quantitative benchmarking practices, published in the last 10 years. It brings together many fine-grained issues in the design and application of benchmarks (such as biases in dataset creation, inadequate documentation, data contamination, and failures to distinguish signal from noise) with broader sociotechnical issues (such as an over-focus on evaluating text-based AI models according to one-time testing logic that fails to account for how AI models are increasingly multimodal and interact with humans and other technical systems). Our review also highlights a series of systemic flaws in current benchmarking practices, such as misaligned incentives, construct validity issues, unknown unknowns, and problems with the gaming of benchmark results. Furthermore, it underscores how benchmark practices are fundamentally shaped by cultural, commercial and competitive dynamics that often prioritise state-of-the-art performance at the expense of broader societal concerns. By providing an overview of risks associated with existing benchmarking procedures, we problematise disproportionate trust placed in benchmarks and contribute to ongoing efforts to improve the accountability and relevance of quantitative AI benchmarks within the complexities of real-world scenarios.