mlsanityData to 14 Oct 2026

Efficient Evaluation of LLM Performance with Statistical Guarantees

arXiv 2601.20251 ↗First posted 28 Jan 2026stat.ML cs.LGcited by 18

Not on the map. Why?

Abstract (arXiv)

Exhaustively evaluating many large language models (LLMs) on a large suite of benchmarks is expensive. We cast benchmarking as finite-population inference and, under a fixed query budget, seek tight confidence intervals (CIs) for model accuracy with valid frequentist coverage. We propose Factorized Active Querying (FAQ), which (a) leverages historical information through a Bayesian factor model; (b) adaptively selects questions using a hybrid variance-reduction/active-learning sampling policy; and (c) maintains validity through Proactive Active Inference -- a finite-population extension of active inference (Zrnic & Cand`es, 2024) that enables direct question selection while preserving coverage. With negligible overhead cost, FAQ delivers up to 5× effective sample size gains over strong baselines on two benchmark suites, across varying historical-data missingness levels: this means that it matches the CI width of uniform sampling while using up to 5× fewer queries. We release our source code and our curated datasets to support reproducible evaluation and future research.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Builds on 31 of its 51 references that are in the map. First posted on arXiv 28 Jan 2026.

Builds on

31 of 51 references.

References · 31

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  4. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  5. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  6. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
  7. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  8. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
  9. 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 310
  10. 2023Large Language Models in Finance: A Surveycited by 226
  11. 2023MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoningcited by 201
  12. 2025Preference Leakage: A Contamination Problem in LLM-as-a-judgecited by 125
19 more
  1. 2024Limitations of the LLM-as-a-Judge Approach for Evaluating LLM Outputs in Expert Knowledge Taskscited by 106
  2. 2024Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Followingcited by 102
  3. 2023Anchor Points: Benchmarking Models with Much Fewer Examplescited by 86
  4. 2024MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Modelscited by 81
  5. 2024Open-LLM-Leaderboard: From Multi-choice to Open-style Questions for LLMs Evaluation, Benchmark, and Arenacited by 76
  6. 2024A Survey on Large Language Models for Critical Societal Domains: Finance, Healthcare, and Lawcited by 74
  7. 2023Efficient Benchmarking of Language Modelscited by 71
  8. 2024Active Statistical Inferencecited by 50
  9. 2024Can Unconfident LLM Annotations Be Used for Confident Conclusions?cited by 49
  10. 2024Active Evaluation Acquisition for Efficient LLM Benchmarkingcited by 35
  11. 2025How Benchmark Prediction from Fewer Data Misses the Markcited by 28
  12. 2025How to Select Datapoints for Efficient Human Evaluation of NLG Models?cited by 23
  13. 2025Cost-Optimal Active AI Model Evaluationcited by 19
  14. 2023Active Learning and Bayesian Optimization: a Unified Perspective to Learn with a Goalcited by 17
  15. 2024On Speeding Up Language Model Evaluationcited by 15
  16. 2025Anatomy of a Machine Learning Ecosystem: 2 Million Models on Hugging Facecited by 12
  17. 2025LeMAJ (Legal LLM-as-a-Judge): Bridging Legal Reasoning and LLM Evaluationcited by 10
  18. 2025Efficient Evaluation of Large Language Models via Collaborative Filteringcited by 5
  19. 2025LongReasonArena: A Long Reasoning Benchmark for Large Language Modelscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 18.

18 papers.

Sort:

By area:

  1. 2026Towards Reliable LLM Evaluation: Correcting the Winner's Curse in Adaptive Benchmarkingcited by 7
  2. 2026Sanity Checks for Agentic Data Sciencecited by 4
  3. 2026How Many Iterations to Jailbreak? Dynamic Budget Allocation for Multi-Turn LLM Evaluationcited by 4
  4. 2026Valid Best-Model Identification for LLM Evaluation via Low-Rank Factorizationcited by 4
  5. 2026How Many Tasks Are Enough for Agent Benchmark Decisions? A Replay Analysis of Public LLM Agent Benchmarkscited by 4
  6. 2026Efficient Sequential Evaluation of Large Language Modelscited by 4
  7. 2026EarlyEval: Cheaper Agent Evaluation via Early Outcome Predictioncited by 3
  8. 2026Measuring Five-Nines Reliability: Sample-Efficient LLM Evaluation in Saturated Benchmarkscited by 2
  9. 2026Metric Match: A Subset Selection Approach to Evaluating LLM Judge Reliabilitycited by 1
  10. 2026ParEvalLayer: When Partial LLM-Agent Evaluations Support a Decisioncited by 1
  11. 2026An Interpretable and Scalable Framework for Evaluating Large Language Modelscited by 0
  12. 2026Hypothesis Testing with Conditional Queries: Learnability and the Value of Interactioncited by 0
6 more
  1. 2026Human-Anchored Factuality Evaluation with Strategic Annotationcited by 0
  2. 2026Prediction-Powered Smoothing and Validation for Disaggregated AI Evaluationcited by 0
  3. 2026Speculative Evaluation of Stochastic LLMscited by 0
  4. 2026Sharp Limits for Honest Uncertainty in Hard-Budget Repeated Evaluationcited by 0
  5. 2026When Does a Skill Add Value? Task-Conditional Gain Prediction for Selective Skill Usecited by 0
  6. 2026Are We Testing Too Much? A Survey of Benchmark Compression for Efficient Evaluation of Large Language Modelscited by 0