mlsanityData to 7 Oct 2026

Reliable and Efficient Amortized Model-based Evaluation

arXiv 2503.13335 ↗First posted 17 Mar 2025cs.CL cs.AI cs.LGcited by 29

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Comprehensive evaluations of language models (LM) during both development and deployment phases are necessary because these models possess numerous capabilities (e.g., mathematical reasoning, legal support, or medical diagnostic) as well as safety risks (e.g., racial bias, toxicity, or misinformation). The average score across a wide range of benchmarks provides a signal that helps guide the use of these LMs in practice. Currently, holistic evaluations are costly due to the large volume of benchmark questions, making frequent evaluations impractical. A popular attempt to lower the cost is to compute the average score on a subset of the benchmark. This approach, unfortunately, often renders an unreliable measure of LM performance because the average score is often confounded with the difficulty of the questions in the benchmark subset. Item response theory (IRT) was designed to address this challenge, providing a reliable measurement by careful controlling for question difficulty. Unfortunately, question difficulty is expensive to estimate. Facing this challenge, we train a model that predicts question difficulty from its content, enabling a reliable measurement at a fraction of the cost. In addition, we leverage this difficulty predictor to further improve the evaluation efficiency through training a question generator given a difficulty level. This question generator is essential in adaptive testing, where, instead of using a random subset of the benchmark questions, informative questions are adaptively chosen based on the current estimation of LLM performance. Experiments on 22 common natural language benchmarks and 172 LMs show that this approach is more reliable and efficient compared to current common practice.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 12 of its 31 references that are in the map. First posted on arXiv 17 Mar 2025.

Builds on

12 of 31 references.

References · 12

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2021Explaining Neural Scaling Lawscited by 353
  5. 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 295
  6. 2024Observational Scaling Laws and the Predictability of Language Model Performancecited by 128
  7. 2023Anchor Points: Benchmarking Models with Much Fewer Examplescited by 84
  8. 2023Efficient Benchmarking of Language Modelscited by 71
  9. 2021Comparing Test Sets with Item Response Theorycited by 55
  10. 2019Learning Latent Parameters without Human Response Patterns: Item Response Theory with Artificial Crowdscited by 40
  11. 2020Variational Item Response Theory: Fast, Accurate, and Expressivecited by 29
  12. 2024Data Efficient Evaluation of Large Language Models and Text-to-Image Models via Adaptive Samplingcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 23.

29 papers.

Sort:

By area:

  1. 2025A Rosetta Stone for AI Benchmarkscited by 29
  2. 2026Agent psychometrics: Task-level performance prediction in agentic coding benchmarkscited by 8
  3. 2025Confidence in Large Language Model Evaluation: A Bayesian Approach to Limited-Sample Challengescited by 7
  4. 2026BRIDGE: Predicting Human Task Completion Time From Model Performancecited by 6
  5. 2025Toward a unified framework for data-efficient evaluation of large language modelscited by 5
  6. 2025Discovering Hierarchical Latent Capabilities of Language Models via Causal Representation Learningcited by 4
  7. 2026Can We Trust Item Response Theory for AI Evaluation?cited by 3
  8. 2026InfoSynth: Information-Guided Benchmark Synthesis for LLMscited by 1
  9. 2026AI Evaluation Should Require Standardized Item-Level Data Releasescited by 1
  10. 2026The Scaling Law of Evaluation Failure: Why Simple Averaging Collapses Under Data Sparsity and Item Difficulty Gaps, and How Item Response Theory Recovers Ground Truth Across Domainscited by 1
  11. 2026Measuring Five-Nines Reliability: Sample-Efficient LLM Evaluation in Saturated Benchmarkscited by 1
  12. 2026Quality Is Not a Safety Proxy Under Quantizationcited by 1
17 more
  1. 2026Offline Preference-Based Trajectory Evaluationcited by 1
  2. 2026Diagnose Before You Compress: Prediction-Independent Bottleneck Witness Refinement for LLM Serving Tracescited by 1
  3. 2026LLM Evaluation on Unseen Questions: Contextual Multidimensional IRT Modelcited by 1
  4. 2025The Self-Execution Benchmark: Measuring LLMs' Attempts to Overcome Their Lack of Self-Executioncited by 0
  5. 2025GPT-5 Model Corrected GPT-4V's Chart Reading Errors, Not Promptingcited by 0
  6. 2026Benchmarking at the Edge of Comprehensioncited by 0
  7. 2026Beyond Fixed Benchmarks and Worst-Case Attacks: Dynamic Boundary Evaluation for Language Modelscited by 0
  8. 2026Correcting test set contamination by spiking the training datacited by 0
  9. 2026Consistent and Distinctive: LLM Benchmark Efficiency via Maximum Independent Set Prompt Selection on Similarity Graphscited by 0
  10. 2026Item Response Scaling Laws: A Measurement Theory Approach for Efficient and Generalizable Neural Scaling Estimationcited by 0
  11. 2026Latent Confidence Alignment for LLM Self-Assessmentcited by 0
  12. 2026Predicting Task Difficulty Without Rolloutscited by 0
  13. 2026Hypothesis Testing with Conditional Queries: Learnability and the Value of Interactioncited by 0
  14. 2026What Makes Software Issue Resolution Tasks Difficult for Agents?cited by 0
  15. 2026Efficient Benchmarking in Production: A Study of an Evolving LLM Agentcited by 0
  16. 2026SecProbe: Adaptive Evaluation of Coding Agents on Cybersecurity Vulnerabilitiescited by 0
  17. 2026Rubric Rewards from Item Response Theorycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Lost in Benchmarks? Rethinking Large Language Model Benchmarking with Item Response Theory
  2. 2025Estimating problem difficulty without ground truth using Large Language Model comparisons
  3. 2025Latency-Response Theory Model: Evaluating Large Language Models via Response Accuracy and Chain-of-Thought Length
  4. 2025SAS-Bench: A Fine-Grained Benchmark for Evaluating Short Answer Scoring with Large Language Models
  5. 2024Evaluating Students' Open-ended Written Responses with LLMs: Using the RAG Framework for GPT-3.5, GPT-4, Claude-3, and Mistral-Large
  6. 2026Same Input, Different Scores: A Multi Model Study on the Inconsistency of LLM Judge
  7. 2026When Consistency Does Not Mean Reliability: Evaluating Local LLM Judges Against Human Ratings
  8. 2025Do LLMs Know They Are Being Tested? Evaluation Awareness and Incentive-Sensitive Failures in GPT-OSS-20B