# Fantastic Bugs and Where to Find Them in AI Benchmarks

arXiv: https://arxiv.org/abs/2511.16842 · HTML: https://mlsanity.com/paper/2511.16842/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 49 references that are in the map.
- First posted on arXiv 20 Nov 2025.

## Builds on (most cited first, 14 of 14)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [Adversarial NLI: A New Benchmark for Natural Language Understanding](https://mlsanity.com/paper/1910.14599/) (2019)
- [An Empirical Study of Example Forgetting during Deep Neural Network Learning](https://mlsanity.com/paper/1812.05159/) (2018)
- [Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks](https://mlsanity.com/paper/2103.14749/) (2021)
- [AmbigQA: Answering Ambiguous Open-domain Questions](https://mlsanity.com/paper/2004.10645/) (2020)
- [AIR-Bench 2024: A Safety Benchmark Based on Risk Categories from Regulations and Policies](https://mlsanity.com/paper/2407.17436/) (2024)
- [Do Large Language Model Benchmarks Test Reliability?](https://mlsanity.com/paper/2502.03461/) (2025)
- [More than Marketing? On the Information Value of AI Benchmarks for Practitioners](https://mlsanity.com/paper/2412.05520/) (2024)
- [No Free Lunch: The Hazards of Over-Expressive Representations in Anomaly Detection](https://mlsanity.com/paper/2306.07284/) (2023)
- [No Free Lunch But A Cheaper Supper: A General Framework for Streaming Anomaly Detection](https://mlsanity.com/paper/1909.06927/) (2019)
- [Representation Learning in Anomaly Detection: Successes, Limits and a Grand Challenge](https://mlsanity.com/paper/2307.11085/) (2023)

## Built on by (most cited first, 11 of 11)

- [ProbeLLM: Automating Principled Diagnosis of LLM Failures](https://mlsanity.com/paper/2602.12966/) (2026)
- [WirelessAgent++: Automated Agentic Workflow Design and Benchmarking for Wireless Networks](https://mlsanity.com/paper/2603.00501/) (2026)
- [AI Cartography: Mapping the Latent Landscape of AI Benchmark Ecosystems](https://mlsanity.com/paper/2605.25272/) (2026)
- [Efficient Detection of Bad Benchmark Items with Novel Scalability Coefficients](https://mlsanity.com/paper/2603.24999/) (2026)
- [Are Near-Tied LLM Rankings Robust to Family-DIF-Guided Benchmark Recomposition?](https://mlsanity.com/paper/2609.00482/) (2026)
- [Why Do Safety Guardrails Degrade Across Languages?](https://mlsanity.com/paper/2605.17173/) (2026)
- [Welfare, Improvability, and Variance: A Principal-Agent Approach to Optimal Benchmark Item Aggregation](https://mlsanity.com/paper/2605.30916/) (2026)
- [LEVANTE-bench: Multi-Scale Comparison of VLMs to Children Using Cognitive Tasks (or, "Is Your VLM Smarter Than a 5th Grader?")](https://mlsanity.com/paper/2606.05497/) (2026)
- [API Benchmark Scores Do Not Reliably Transfer to Chatbot Interfaces](https://mlsanity.com/paper/2609.08861/) (2026)
- [Are Benchmarks Reliable? Toward Structural Diagnosis via Item-Level Capability Boundaries](https://mlsanity.com/paper/2609.33196/) (2026)
- [Auditing VLM Benchmarks from Existing Evaluation Runs](https://mlsanity.com/paper/2610.13548/) (2026)

## Abstract (arXiv)

Benchmarks are pivotal in driving AI progress, and invalid benchmark questions frequently undermine their reliability. Manually identifying and correcting errors among thousands of benchmark questions is not only infeasible but also a critical bottleneck for reliable evaluation. In this work, we introduce a framework for systematic benchmark revision that leverages statistical analysis of response patterns to flag potentially invalid questions for further expert review. Our approach builds on a core assumption commonly used in AI evaluations that the mean score sufficiently summarizes model performance. This implies a unidimensional latent construct underlying the measurement experiment, yielding expected ranges for various statistics for each item. When empirically estimated values for these statistics fall outside the expected range for an item, the item is more likely to be problematic. Across nine widely used benchmarks, our method guides expert review to identify problematic questions with up to 84% precision. In addition, we introduce an LLM-judge first pass to review questions, further reducing human effort. Together, these components provide an efficient and scalable framework for systematic benchmark revision.