# On Challenges in Evaluation Methodology: A Review

arXiv: https://arxiv.org/abs/2604.25923 · HTML: https://mlsanity.com/paper/2604.25923/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 304 references that are in the map.
- First posted on arXiv 1 Apr 2026.

## Builds on (most cited first, 9 of 9)

- [Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks](https://mlsanity.com/paper/2103.14749/) (2021)
- [Measuring what Matters: Construct Validity in Large Language Model Benchmarks](https://mlsanity.com/paper/2511.04703/) (2025)
- [Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Research](https://mlsanity.com/paper/2112.01716/) (2021)
- [The Leaderboard Illusion](https://mlsanity.com/paper/2504.20879/) (2025)
- [Position: Evaluating Generative AI Systems Is a Social Science Measurement Challenge](https://mlsanity.com/paper/2502.00561/) (2025)
- [Toward an Evaluation Science for Generative AI Systems](https://mlsanity.com/paper/2503.05336/) (2025)
- [Lost in Benchmarks? Rethinking Large Language Model Benchmarking with Item Response Theory](https://mlsanity.com/paper/2505.15055/) (2025)
- [LLM-Evaluation Tropes: Perspectives on the Validity of LLM-Evaluations](https://mlsanity.com/paper/2504.19076/) (2025)
- [Are Domain Generalization Benchmarks with Accuracy on the Line Misspecified?](https://mlsanity.com/paper/2504.00186/) (2025)

## Abstract (arXiv)

Recent advances in language models have prompted a growing body of work that questions prevailing evaluation practices. However, many such critiques have already been discussed in natural language processing (NLP), a field with a long history of methodological reflection on evaluation. We conduct a scoping review spanning four decades and develop a taxonomy of 22 recurring concerns. For each concern, we examine how recently it was discussed in the reviewed literature and whether it receives explicit attention in institutional checklists, finding that some concerns have become less visible in literature and many lack coverage in these checklists. Our work provides an overview of the major concerns in evaluation methodology that have shaped the field and identifies potential targets for renewed investigation and field-level intervention by the research community.