mlsanityData to 14 Oct 2026

Lost in Benchmarks? Rethinking Large Language Model Benchmarking with Item Response Theory

arXiv 2505.15055 ↗First posted 21 May 2025cs.CLcited by 45

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

The evaluation of large language models (LLMs) via benchmarks is widespread, yet inconsistencies between different leaderboards and poor separability among top models raise concerns about their ability to accurately reflect authentic model capabilities. This paper provides a critical analysis of benchmark effectiveness, examining mainstream prominent LLM benchmarks using results from diverse models. We first propose Pseudo-Siamese Network for Item Response Theory (PSN-IRT), an enhanced Item Response Theory framework that incorporates a rich set of item parameters within an IRT-grounded architecture. PSN-IRT can be utilized for accurate and reliable estimations of item characteristics and model abilities. Based on PSN-IRT, we conduct extensive analysis on 11 LLM benchmarks comprising 41,871 items, revealing significant and varied shortcomings in their measurement quality. Furthermore, we demonstrate that leveraging PSN-IRT is able to construct smaller benchmarks while maintaining stronger alignment with human preference.

Key facts. Cited by 45 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 13 of its 47 references that are in the map. First posted on arXiv 21 May 2025.

Builds on

13 of 47 references.

References · 13

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2024Qwen2.5 Technical Reportcited by 4,942
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  4. 2021Program Synthesis with Large Language Modelscited by 3,637
  5. 2023Mistral 7Bcited by 3,164
  6. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  7. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  8. 2024Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligencecited by 68
  9. 2025Large Language Model Psychometrics: A Systematic Review of Evaluation, Validation, and Enhancementcited by 45
  10. 2024Chinese SimpleQA: A Chinese Factuality Evaluation for Large Language Modelscited by 43
  11. 2025General Scales Unlock AI Evaluation with Explanatory and Predictive Powercited by 31
  12. 2024Do These LLM Benchmarks Agree? Fixing Benchmark Evaluation with BenchBenchcited by 27
1 more
  1. 2024100 instances is all you need: predicting the success of a new LLM on unseen data by testing on a few instancescited by 24

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 39.

45 papers.

Sort:

By area:

  1. 2025A Rosetta Stone for AI Benchmarkscited by 36
  2. 2026Mem2Evolve: Towards Self-Evolving Agents via Co-Evolutionary Capability Expansion and Experience Distillationcited by 12
  3. 2026Efficient Benchmarking of AI Agentscited by 10
  4. 2026Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimizationcited by 9
  5. 2025Toward a unified framework for data-efficient evaluation of large language modelscited by 5
  6. 2026PACE: A Proxy for Agentic Capability Evaluationcited by 5
  7. 2025JE-IRT: A Geometric Lens on LLM Abilities through Joint Embedding Item Response Theorycited by 4
  8. 2026Can We Trust Item Response Theory for AI Evaluation?cited by 4
  9. 2025Position: AI Evaluations Should be Grounded on a Theory of Capabilitycited by 2
  10. 2025Identity-Link IRT for Label-Free LLM Evaluation: Preserving Additivity in TVD-MI Scorescited by 2
  11. 2026The Subjectivity of Monoculturecited by 2
  12. 2026Rich Insights from Cheap Signals: Efficient Evaluations via Tensor Factorizationcited by 2
33 more
  1. 2026Reasoning Models Don't Just Think Longer, They Move Differentlycited by 2
  2. 2026Extending Item Response Theory for Efficient and Meaningful Multilingual Evaluationcited by 2
  3. 2025Measuring Competency, Not Performance: Item-Aware Evaluation Across Medical Benchmarkscited by 1
  4. 2026Benchmarking at the Edge of Comprehensioncited by 1
  5. 2026Ranking Reasoning LLMs under Test-Time Scalingcited by 1
  6. 2026BenchBrowser: Retrieving Evidence for Evaluating Benchmark Validitycited by 1
  7. 2026Beyond Scores: Diagnostic LLM Evaluation via Fine-Grained Abilitiescited by 1
  8. 2026The Scaling Law of Evaluation Failure: Why Simple Averaging Collapses Under Data Sparsity and Item Difficulty Gaps, and How Item Response Theory Recovers Ground Truth Across Domainscited by 1
  9. 2026Machine Psychometrics: A Mathematical Psychology of Artificial Intelligencecited by 1
  10. 2026Consistent and Distinctive: LLM Benchmark Efficiency via Maximum Independent Set Prompt Selection on Similarity Graphscited by 1
  11. 2026DualEval: Joint Model-Item Calibration for Unified LLM Evaluationcited by 1
  12. 2026The yes-no bias of large language models reflects answer order and wording, not shifts in moral judgmentcited by 1
  13. 2026Laplace-PSN-IRT: Uncertainty Quantification for Neural Item Response Theory Models of LLM Benchmarkscited by 1
  14. 2026LLM Evaluation on Unseen Questions: Contextual Multidimensional IRT Modelcited by 1
  15. 2026Are Near-Tied LLM Rankings Robust to Family-DIF-Guided Benchmark Recomposition?cited by 1
  16. 2026Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluationcited by 1
  17. 2026Anytime-Valid LLM Leaderboards via Benchmark-weighted and Block-Factorized e-Processescited by 1
  18. 2026Benchmark Health Index: A Systematic Framework for Benchmarking the Benchmarks of LLMscited by 0
  19. 2026RankLLM: Weighted Ranking of LLMs by Quantifying Question Difficultycited by 0
  20. 2026DepthCharge: A Domain-Agnostic Framework for Measuring Depth-Dependent Knowledge in Large Language Modelscited by 0
  21. 2026On Challenges in Evaluation Methodology: A Reviewcited by 0
  22. 2026Estimating LLM Grading Ability and Response Difficulty in Automatic Short Answer Grading via Item Response Theorycited by 0
  23. 2026Beyond Fixed Benchmarks and Worst-Case Attacks: Dynamic Boundary Evaluation for Language Modelscited by 0
  24. 2026Does Capability Transfer to Subjective Behavior -- and Would Our Instruments Tell Us? A Self-Evolving, Trust-by-Construction Evaluation Paradigmcited by 0
  25. 2026Welfare, Improvability, and Variance: A Principal-Agent Approach to Optimal Benchmark Item Aggregationcited by 0
  26. 2026Deployment-Centered Evaluation: Predicting Query-Level Rejection Risk in a Clinical LLM Systemcited by 0
  27. 2026Latent Confidence Alignment for LLM Self-Assessmentcited by 0
  28. 2026EduArt: An educational-level benchmark for evaluating art history knowledge in large language modelscited by 0
  29. 2026Efficient SWE Agent Benchmarking via Trajectory-Aware Evaluationcited by 0
  30. 2026Coding Agents Have Converged: Why the SWE-bench Leaderboard Can No Longer Order Its Top Entries, and What to Measure Insteadcited by 0
  31. 2026Large-scale factor analysis shows machine intelligence is only partially interpretablecited by 0
  32. 2026cua-speedrun: Standardized Benchmarking of the Speed of Computer-Use Agentscited by 0
  33. 2026EvalBuilder: Can LLMs Build Efficient Benchmarks?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026An Interpretable and Scalable Framework for Evaluating Large Language Models
  2. 2026Item Response Scaling Laws: A Measurement Theory Approach for Efficient and Generalizable Neural Scaling Estimation
  3. 2026Latent Performance Profiling of Large Language Models
  4. 2024Evaluating Large Language Models with Human Feedback: Establishing a Swedish Benchmark
  5. 2025Learning Compact Representations of LLM Abilities via Item Response Theory
  6. 2025Reliable and Efficient Amortized Model-based Evaluation
  7. 2024metabench -- A Sparse Benchmark of Reasoning and Knowledge in Large Language Models
  8. 2025Latency-Response Theory Model: Evaluating Large Language Models via Response Accuracy and Chain-of-Thought Length