Are We Testing Too Much? A Survey of Benchmark Compression for Efficient Evaluation of Large Language Models
Not on the map. Why?
Abstract (arXiv)
Benchmarks are essential for evaluating and improving large language models (LLMs). Yet many widely used benchmarks are redundant, making evaluation unnecessarily expensive. Benchmark compression lowers this cost by estimating full-benchmark scores and rankings from a partial evaluation. Research in this area has grown rapidly since 2024, but to our knowledge no survey has systematically organized its methods and findings. We review 119 papers to explain how methods choose what to evaluate, recover full-benchmark results, and verify their reliability. From this analysis, we develop a taxonomy covering compression within individual benchmarks, across benchmark suites, and during adaptive evaluation, together with work on redundancy and reliability. We compare their information needs, total costs, reliability checks, and limitations. Methods reduce evaluation costs by selecting representative subsets, using cheaper or reusable information, and testing adaptively, but risk poor coverage, inaccurate estimates, and sequential delays. The resulting taxonomy and practical guide help researchers choose, validate, and deploy methods for more efficient and reliable LLM evaluation.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 94 of its 121 references that are in the map. First posted on arXiv 13 Oct 2026.
Builds on
94 of 121 references.
References · 94
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2020Measuring Massive Multitask Language Understandingcited by 7,963
- 2024A Survey on LLM-as-a-Judgecited by 1,423
- 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 310
- 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systemscited by 265
- 2024Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluationscited by 148
- 2024Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreementcited by 105
- 2023Anchor Points: Benchmarking Models with Much Fewer Examplescited by 86
- 2024A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendationscited by 85
- 2024Efficient multi-prompt evaluation of LLMscited by 73
- 2024AutoEval Done Right: Using Synthetic Data for Model Evaluationcited by 57
- 2025Large Language Model Psychometrics: A Systematic Review of Evaluation, Validation, and Enhancementcited by 45
38 more
- 2024Active Evaluation Acquisition for Efficient LLM Benchmarkingcited by 35
- 2026Stop Comparing LLM Agents Without Disclosing the Harnesscited by 34
- 2025Reliable and Efficient Amortized Model-based Evaluationcited by 31
- 2025How to Correctly Report LLM-as-a-Judge Evaluationscited by 30
- 2025How Benchmark Prediction from Fewer Data Misses the Markcited by 28
- 2025Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarkscited by 28
- 2026Noisy but Valid: Robust Statistical Evaluation of LLMs with Imperfect Judgescited by 24
- 2024Improving Model Evaluation using SMART Filtering of Benchmark Datasetscited by 21
- 2024Label-Efficient Model Selection for Text Generationcited by 19
- 2025Cost-Optimal Active AI Model Evaluationcited by 19
- 2024Sample-Efficient Human Evaluation of Large Language Models via Maximum Discrepancy Competitioncited by 18
- 2026Efficient Evaluation of LLM Performance with Statistical Guaranteescited by 18
- 2024AcTracer: Active Testing of Large Language Model via Multi-Stage Samplingcited by 17
- 2024On Speeding Up Language Model Evaluationcited by 15
- 2024ONEBench to Test Them All: Sample-Level Benchmarking Over Open-Ended Capabilitiescited by 15
- 2026Benchmark^2: Systematic Evaluation of LLM Benchmarkscited by 13
- 2024Data Efficient Evaluation of Large Language Models and Text-to-Image Models via Adaptive Samplingcited by 12
- 2025Beyond One-Size-Fits-All: Tailored Benchmarks for Efficient Evaluationcited by 12
- 2025Cer-Eval: Certifiable and Cost-Efficient Evaluation Framework for LLMscited by 11
- 2025Adaptive Prediction-Powered AutoEval with Reliability and Efficiency Guaranteescited by 11
- 2025ReliableEval: A Recipe for Stochastic LLM Evaluation via Method of Momentscited by 11
- 2025Scales++: Compute Efficient Evaluation Subset Selection with Cognitive Scales Embeddingscited by 11
- 2026Agent psychometrics: Task-level performance prediction in agentic coding benchmarkscited by 11
- 2026DatBench: Discriminative, Faithful, and Efficient VLM Evaluationscited by 10
- 2024Arena-Lite: Efficient and Reliable Large Language Model Evaluation via Tournament-Based Direct Comparisonscited by 9
- 2025From Benchmarks to Skills: Low-Rank Factors for LLM Evaluationcited by 9
- 2025EffiEval: Efficient and Generalizable Model Evaluation via Capability Coverage Maximizationcited by 9
- 2025How Reliable is Language Model Micro-Benchmarking?cited by 9
- 2026When LLM Judge Scores Look Good but Best-of-N Decisions Failcited by 9
- 2025MiniLongBench: The Low-cost Long Context Understanding Benchmark for Large Language Modelscited by 8
- 2026Neural Chain-of-Thought Search: Searching the Optimal Reasoning Path to Enhance Large Language Modelscited by 8
- 2026SparseEval: Efficient Evaluation of Large Language Models by Sparse Optimizationcited by 8
- 2024Regression for the Mean: Auto-Evaluation and Inference with Few Labels through Post-hoc Regressioncited by 7
- 2025Accelerating Unbiased LLM Evaluation via Synthetic Feedbackcited by 7
- 2026Resolution Diagnostics for Paired LLM Evaluationcited by 7
- 2025SimBA: Simplifying Benchmark Analysis Using Performance Matrices Alonecited by 6
- 2025Efficient Evaluation of Large Language Models via Collaborative Filteringcited by 5
- 2025DISCO: Diversifying Sample Condensation for Efficient Model Evaluationcited by 5
50 most cited of 94 All 94 (JSON)
Built on by
Not cited yet.