# Stop Guessing When to Stop Testing: Efficient Model Evaluation with Just Enough Data

arXiv: https://arxiv.org/abs/2607.08522 · HTML: https://mlsanity.com/paper/2607.08522/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 32 references that are in the map.
- First posted on arXiv 9 Jul 2026.

## Builds on (most cited first, 14 of 14)

- [A Diagram Is Worth A Dozen Images](https://mlsanity.com/paper/1603.07396/) (2016)
- [From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline](https://mlsanity.com/paper/2406.11939/) (2024)
- [VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Models](https://mlsanity.com/paper/2407.11691/) (2024)
- [tinyBenchmarks: evaluating LLMs with fewer examples](https://mlsanity.com/paper/2402.14992/) (2024)
- [Anchor Points: Benchmarking Models with Much Fewer Examples](https://mlsanity.com/paper/2309.08638/) (2023)
- [Active Testing: Sample-Efficient Model Evaluation](https://mlsanity.com/paper/2103.05331/) (2021)
- [Efficient multi-prompt evaluation of LLMs](https://mlsanity.com/paper/2405.17202/) (2024)
- [Efficient Prompt Optimization Through the Lens of Best Arm Identification](https://mlsanity.com/paper/2402.09723/) (2024)
- [Benchmarking Large Language Model Capabilities for Conditional Generation](https://mlsanity.com/paper/2306.16793/) (2023)
- [Label-Efficient Model Selection for Text Generation](https://mlsanity.com/paper/2402.07891/) (2024)
- [A Framework for Efficient Model Evaluation through Stratification, Sampling, and Estimation](https://mlsanity.com/paper/2406.07320/) (2024)
- [Near-Optimal Non-Parametric Sequential Tests and Confidence Sequences with Possibly Dependent Observations](https://mlsanity.com/paper/2212.14411/) (2022)
- [Can We Predict Performance of Large Models across Vision-Language Tasks?](https://mlsanity.com/paper/2410.10112/) (2024)
- [AdaStop: adaptive statistical testing for sound comparisons of Deep RL agents](https://mlsanity.com/paper/2306.10882/) (2023)

## Built on by (most cited first, 2 of 2)

- [Rank Confidence Sequences:Anytime-valid Leaderboards](https://mlsanity.com/paper/2609.32211/) (2026)
- [Are We Testing Too Much? A Survey of Benchmark Compression for Efficient Evaluation of Large Language Models](https://mlsanity.com/paper/2610.18248/) (2026)

## Abstract (arXiv)

The inherent rigidity of fixed-size benchmarks makes them an inefficient tool for model evaluation. Diverse evaluation objectives, including model ranking, model selection and testing throughout development, demand varying levels of statistical power. The mismatch between fixed sample sizes and these diverse needs results in either excessive computational cost or compromised reliability - a critical concern for model evaluation. To overcome these limitations, we call for adoption of sequential testing in our field. We provide an adaptive evaluation framework, that provides a principled way to navigate the trade-off between efficiency and reliability in model evaluation. Our framework combines the established statistical paradigm of sequential testing with stopping criteria tailored to common evaluation needs such as diminishing returns detection, and minimum detectable effect size. We demonstrate its ability to adaptively manage the efficiency-reliability trade-off on the Open VLM Leaderboard, including, for example, a 80% reduction in computational cost compared to fixed-size evaluation (with a 2.5-point CI width allowance) while maintaining statistical significance.