# Confident Rankings with Fewer Items: Adaptive LLM Evaluation with Continuous Scores

arXiv: https://arxiv.org/abs/2601.13885 · HTML: https://mlsanity.com/paper/2601.13885/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 33 references that are in the map.
- First posted on arXiv 20 Jan 2026.

## Builds on (most cited first, 11 of 11)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Efficient Attentions for Long Document Summarization](https://mlsanity.com/paper/2104.02112/) (2021)
- [tinyBenchmarks: evaluating LLMs with fewer examples](https://mlsanity.com/paper/2402.14992/) (2024)
- [Comparing Test Sets with Item Response Theory](https://mlsanity.com/paper/2106.00840/) (2021)
- [Learning Latent Parameters without Human Response Patterns: Item Response Theory with Artificial Crowds](https://mlsanity.com/paper/1908.11421/) (2019)
- [metabench -- A Sparse Benchmark of Reasoning and Knowledge in Large Language Models](https://mlsanity.com/paper/2407.12844/) (2024)
- [Overview of the BioLaySumm 2024 Shared Task on the Lay Summarization of Biomedical Research Articles](https://mlsanity.com/paper/2408.08566/) (2024)
- [β3-IRT: A New Item Response Model and its Applications](https://mlsanity.com/paper/1903.04016/) (2019)

## Built on by (most cited first, 5 of 5)

- [Efficient Benchmarking Is Just Feature Selection and Multiple Regression](https://mlsanity.com/paper/2605.25773/) (2026)
- [Knowing When to Stop: Bayesian Optimal Stopping for LLM Evaluations](https://mlsanity.com/paper/2608.14425/) (2026)
- [Dynamically Allocating Evaluation Effort for Model Ranking](https://mlsanity.com/paper/2608.03437/) (2026)
- [Large Language Models Can Follow Instructions, But Not Many at Once: Phase Transitions in Compositional Constraint Satisfaction](https://mlsanity.com/paper/2608.12426/) (2026)
- [Are We Testing Too Much? A Survey of Benchmark Compression for Efficient Evaluation of Large Language Models](https://mlsanity.com/paper/2610.18248/) (2026)

## Abstract (arXiv)

Computerized Adaptive Testing (CAT) has proven effective for efficient LLM evaluation on multiple-choice benchmarks, but modern LLM evaluation increasingly relies on generation tasks where outputs are scored continuously rather than marked correct/incorrect. We present a principled extension of IRT-based adaptive testing to continuous bounded scores (ROUGE, BLEU, LLM-as-a-Judge) by replacing the Bernoulli response distribution with a heteroskedastic normal distribution. Building on this, we introduce an uncertainty aware ranker with adaptive stopping criteria that achieves reliable model ranking while testing as few items and as cheaply as possible. We validate our method on five benchmarks spanning n-gram-based, embedding-based, and LLM-as-judge metrics. Our method improves ranking correlation by 0.13 τ over random sampling and has 99% accuracy on confident predictions while using 2% of the items after a one-time calibration step.