# Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks

arXiv: https://arxiv.org/abs/2511.04689 · HTML: https://mlsanity.com/paper/2511.04689/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 41 references that are in the map.
- First posted on arXiv 26 Oct 2025.

## Builds on (most cited first, 12 of 12)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [tinyBenchmarks: evaluating LLMs with fewer examples](https://mlsanity.com/paper/2402.14992/) (2024)
- [Are We Done with MMLU?](https://mlsanity.com/paper/2406.04127/) (2024)
- [Do Large Language Model Benchmarks Test Reliability?](https://mlsanity.com/paper/2502.03461/) (2025)
- [metabench -- A Sparse Benchmark of Reasoning and Knowledge in Large Language Models](https://mlsanity.com/paper/2407.12844/) (2024)
- [Position: AI Evaluation Should Learn from How We Test Humans](https://mlsanity.com/paper/2306.10512/) (2023)
- [Fluid Language Model Benchmarking](https://mlsanity.com/paper/2509.11106/) (2025)
- [Evaluating Large Language Models with Psychometrics](https://mlsanity.com/paper/2406.17675/) (2024)

## Built on by (most cited first, 28 of 28)

- [A Rosetta Stone for AI Benchmarks](https://mlsanity.com/paper/2512.00193/) (2025)
- [EvolveRouter: Co-Evolving Routing and Prompt for Multi-Agent Question Answering](https://mlsanity.com/paper/2604.05149/) (2026)
- [Agent psychometrics: Task-level performance prediction in agentic coding benchmarks](https://mlsanity.com/paper/2604.00594/) (2026)
- [Why Semantic Entropy Fails: Geometry-Aware and Calibrated Uncertainty for Policy Optimization](https://mlsanity.com/paper/2605.21801/) (2026)
- [Auditing LLM Benchmarks with Item Response Theory](https://mlsanity.com/paper/2605.30504/) (2026)
- [SAGE: Answer-Conditioned Uncertainty Targets for Verbal Uncertainty Alignment](https://mlsanity.com/paper/2606.11512/) (2026)
- [Every Eval Ever: A Unifying Schema and Community Repository for AI Evaluation Results](https://mlsanity.com/paper/2606.14516/) (2026)
- [EarlyEval: Cheaper Agent Evaluation via Early Outcome Prediction](https://mlsanity.com/paper/2609.02783/) (2026)
- [Cost-Efficient Estimation of General Abilities Across Benchmarks](https://mlsanity.com/paper/2604.01418/) (2026)
- [Extending Item Response Theory for Efficient and Meaningful Multilingual Evaluation](https://mlsanity.com/paper/2606.15643/) (2026)
- [Knowing When to Stop: Bayesian Optimal Stopping for LLM Evaluations](https://mlsanity.com/paper/2608.14425/) (2026)
- [NAAMSE: Framework for Evolutionary Security Evaluation of Agents](https://mlsanity.com/paper/2602.07391/) (2026)
- [DualEval: Joint Model-Item Calibration for Unified LLM Evaluation](https://mlsanity.com/paper/2606.26429/) (2026)
- [Every Wrong Answer Counts: Option-Level Psychometrics for LLM Multiple-Choice Benchmarks](https://mlsanity.com/paper/2608.02966/) (2026)
- [RankLLM: Weighted Ranking of LLMs by Quantifying Question Difficulty](https://mlsanity.com/paper/2602.12424/) (2026)
- [DepthCharge: A Domain-Agnostic Framework for Measuring Depth-Dependent Knowledge in Large Language Models](https://mlsanity.com/paper/2603.23514/) (2026)
- [Growing Pains: Extensible and Efficient LLM Benchmarking Via Fixed Parameter Calibration](https://mlsanity.com/paper/2604.12843/) (2026)
- [Beyond Fixed Benchmarks and Worst-Case Attacks: Dynamic Boundary Evaluation for Language Models](https://mlsanity.com/paper/2605.06213/) (2026)
- [Capturing LLM Capabilities via Evidence-Calibrated Query Clustering](https://mlsanity.com/paper/2605.17110/) (2026)
- [The Masked Advantage: Uncovering Local-Language Access to Cultural Knowledge in LLMs](https://mlsanity.com/paper/2606.07422/) (2026)
- [FailureScope: Cross-Regime Behavioral Diagnosis of Language Model Weaknesses](https://mlsanity.com/paper/2606.09878/) (2026)
- [CoT-Core: Accelerating LLM Evaluation via CoT-Aware Coreset Selection](https://mlsanity.com/paper/2608.00014/) (2026)
- [Large Language Models Can Follow Instructions, But Not Many at Once: Phase Transitions in Compositional Constraint Satisfaction](https://mlsanity.com/paper/2608.12426/) (2026)
- [Frontier AI Forecasting Has a Measurement Problem: An Audit of Progress Evidence](https://mlsanity.com/paper/2608.14903/) (2026)
- [Evaluation-as-Search: Adaptive Discovery of Grounding Failures in Meeting Assistants](https://mlsanity.com/paper/2608.20392/) (2026)
- [Zipbench: Low-Cost Framework for Compressing Comprehensive Benchmarks of Large Language Models](https://mlsanity.com/paper/2609.12475/) (2026)
- [Benchmarking Psychological Dynamics in Generative Agents](https://mlsanity.com/paper/2610.04246/) (2026)
- [Are We Testing Too Much? A Survey of Benchmark Compression for Efficient Evaluation of Large Language Models](https://mlsanity.com/paper/2610.18248/) (2026)

## Abstract (arXiv)

Evaluating large language models (LLMs) typically requires thousands of benchmark items, making the process expensive, slow, and increasingly impractical at scale. Existing evaluation protocols rely on average accuracy over fixed item sets, treating all items as equally informative despite substantial variation in difficulty and discrimination. We introduce ATLAS, an adaptive testing framework based on Item Response Theory (IRT) that estimates model ability using Fisher information-guided item selection. ATLAS reduces the number of required items by up to 90% while maintaining measurement precision. For instance, it matches whole-bank ability estimates using only 41 items (0.157 MAE) on HellaSwag (5,600 items). We further reconstruct accuracy from ATLAS's ability estimates and find that reconstructed accuracies closely match raw accuracies across all five benchmarks, indicating that ability θ preserves the global performance structure. At the same time, θ provides finer discrimination within accuracy-equivalent models: among more than 3,000 evaluated models, 23-31% shift by more than 10 rank positions, and models with identical accuracies receive meaningfully different ability estimates. Code and calibrated item banks are available at https://github.com/Peiyu-Georgia-Li/ATLAS.git.