# Lost in Benchmarks? Rethinking Large Language Model Benchmarking with Item Response Theory

arXiv: https://arxiv.org/abs/2505.15055 · HTML: https://mlsanity.com/paper/2505.15055/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 13 of its 47 references that are in the map.
- First posted on arXiv 21 May 2025.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 13 of 13)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Gemma: Open Models Based on Gemini Research and Technology](https://mlsanity.com/paper/2403.08295/) (2024)
- [From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline](https://mlsanity.com/paper/2406.11939/) (2024)
- [Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence](https://mlsanity.com/paper/2402.09880/) (2024)
- [Large Language Model Psychometrics: A Systematic Review of Evaluation, Validation, and Enhancement](https://mlsanity.com/paper/2505.08245/) (2025)
- [Chinese SimpleQA: A Chinese Factuality Evaluation for Large Language Models](https://mlsanity.com/paper/2411.07140/) (2024)
- [General Scales Unlock AI Evaluation with Explanatory and Predictive Power](https://mlsanity.com/paper/2503.06378/) (2025)
- [Do These LLM Benchmarks Agree? Fixing Benchmark Evaluation with BenchBench](https://mlsanity.com/paper/2407.13696/) (2024)
- [100 instances is all you need: predicting the success of a new LLM on unseen data by testing on a few instances](https://mlsanity.com/paper/2409.03563/) (2024)

## Built on by (most cited first, 30 of 45)

- [A Rosetta Stone for AI Benchmarks](https://mlsanity.com/paper/2512.00193/) (2025)
- [Mem2Evolve: Towards Self-Evolving Agents via Co-Evolutionary Capability Expansion and Experience Distillation](https://mlsanity.com/paper/2604.10923/) (2026)
- [Efficient Benchmarking of AI Agents](https://mlsanity.com/paper/2603.23749/) (2026)
- [Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimization](https://mlsanity.com/paper/2603.08091/) (2026)
- [Toward a unified framework for data-efficient evaluation of large language models](https://mlsanity.com/paper/2510.04051/) (2025)
- [PACE: A Proxy for Agentic Capability Evaluation](https://mlsanity.com/paper/2607.02032/) (2026)
- [JE-IRT: A Geometric Lens on LLM Abilities through Joint Embedding Item Response Theory](https://mlsanity.com/paper/2509.22888/) (2025)
- [Can We Trust Item Response Theory for AI Evaluation?](https://mlsanity.com/paper/2607.15190/) (2026)
- [Position: AI Evaluations Should be Grounded on a Theory of Capability](https://mlsanity.com/paper/2509.19590/) (2025)
- [Identity-Link IRT for Label-Free LLM Evaluation: Preserving Additivity in TVD-MI Scores](https://mlsanity.com/paper/2510.14966/) (2025)
- [The Subjectivity of Monoculture](https://mlsanity.com/paper/2602.24086/) (2026)
- [Rich Insights from Cheap Signals: Efficient Evaluations via Tensor Factorization](https://mlsanity.com/paper/2603.02029/) (2026)
- [Reasoning Models Don't Just Think Longer, They Move Differently](https://mlsanity.com/paper/2605.15454/) (2026)
- [Extending Item Response Theory for Efficient and Meaningful Multilingual Evaluation](https://mlsanity.com/paper/2606.15643/) (2026)
- [Measuring Competency, Not Performance: Item-Aware Evaluation Across Medical Benchmarks](https://mlsanity.com/paper/2509.24186/) (2025)
- [Benchmarking at the Edge of Comprehension](https://mlsanity.com/paper/2602.14307/) (2026)
- [Ranking Reasoning LLMs under Test-Time Scaling](https://mlsanity.com/paper/2603.10960/) (2026)
- [BenchBrowser: Retrieving Evidence for Evaluating Benchmark Validity](https://mlsanity.com/paper/2603.18019/) (2026)
- [Beyond Scores: Diagnostic LLM Evaluation via Fine-Grained Abilities](https://mlsanity.com/paper/2604.12191/) (2026)
- [The Scaling Law of Evaluation Failure: Why Simple Averaging Collapses Under Data Sparsity and Item Difficulty Gaps, and How Item Response Theory Recovers Ground Truth Across Domains](https://mlsanity.com/paper/2605.11205/) (2026)
- [Machine Psychometrics: A Mathematical Psychology of Artificial Intelligence](https://mlsanity.com/paper/2605.23952/) (2026)
- [Consistent and Distinctive: LLM Benchmark Efficiency via Maximum Independent Set Prompt Selection on Similarity Graphs](https://mlsanity.com/paper/2606.01400/) (2026)
- [DualEval: Joint Model-Item Calibration for Unified LLM Evaluation](https://mlsanity.com/paper/2606.26429/) (2026)
- [The yes-no bias of large language models reflects answer order and wording, not shifts in moral judgment](https://mlsanity.com/paper/2607.05552/) (2026)
- [Laplace-PSN-IRT: Uncertainty Quantification for Neural Item Response Theory Models of LLM Benchmarks](https://mlsanity.com/paper/2607.25257/) (2026)
- [LLM Evaluation on Unseen Questions: Contextual Multidimensional IRT Model](https://mlsanity.com/paper/2608.22295/) (2026)
- [Are Near-Tied LLM Rankings Robust to Family-DIF-Guided Benchmark Recomposition?](https://mlsanity.com/paper/2609.00482/) (2026)
- [Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluation](https://mlsanity.com/paper/2609.04298/) (2026)
- [Anytime-Valid LLM Leaderboards via Benchmark-weighted and Block-Factorized e-Processes](https://mlsanity.com/paper/2609.32248/) (2026)
- [Benchmark Health Index: A Systematic Framework for Benchmarking the Benchmarks of LLMs](https://mlsanity.com/paper/2602.11674/) (2026)

## Abstract (arXiv)

The evaluation of large language models (LLMs) via benchmarks is widespread, yet inconsistencies between different leaderboards and poor separability among top models raise concerns about their ability to accurately reflect authentic model capabilities. This paper provides a critical analysis of benchmark effectiveness, examining mainstream prominent LLM benchmarks using results from diverse models. We first propose Pseudo-Siamese Network for Item Response Theory (PSN-IRT), an enhanced Item Response Theory framework that incorporates a rich set of item parameters within an IRT-grounded architecture. PSN-IRT can be utilized for accurate and reliable estimations of item characteristics and model abilities. Based on PSN-IRT, we conduct extensive analysis on 11 LLM benchmarks comprising 41,871 items, revealing significant and varied shortcomings in their measurement quality. Furthermore, we demonstrate that leveraging PSN-IRT is able to construct smaller benchmarks while maintaining stronger alignment with human preference.