mlsanityData to 14 Oct 2026

Scaling Up Active Testing to Large Language Models

arXiv 2508.09093 ↗First posted 12 Aug 2025cs.LG stat.MLcited by 13

Not on the map. Why?

Abstract (arXiv)

Active testing enables label-efficient evaluation of predictive models through careful data acquisition, but it can pose a significant computational cost. We identify cost-saving measures that enable active testing to be scaled up to large language models (LLMs). In particular we show that the surrogate model used to guide data acquisition can be constructed cheaply using in-context learning, does not require updating within an active-testing loop, and can be smaller than the target model. We even find we can make good data-acquisition decisions without making predictions with the target model. As a result we are able to achieve much more accurate evaluations of LLM performance relative to using randomly acquired data. We additionally introduce a bootstrap estimator of evaluation error, which we show to be a useful indicator of how well active testing is working within a single run.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 24 of its 41 references that are in the map. First posted on arXiv 12 Aug 2025.

Builds on

24 of 41 references.

References · 24

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  6. 2025Gemma 3 Technical Reportcited by 1,824
  7. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  8. 2013Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Textscited by 310
  9. 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 310
  10. 2018Hate Speech Dataset from a White Supremacy Forumcited by 198
  11. 2024Evaluating Frontier Models for Dangerous Capabilitiescited by 128
  12. 2023Anchor Points: Benchmarking Models with Much Fewer Examplescited by 86
12 more
  1. 2021Active Testing: Sample-Efficient Model Evaluationcited by 82
  2. 2021On Statistical Bias In Active Learning: How and When To Fix Itcited by 75
  3. 2023In-Context Learning Learns Label Relationships but Is Not Conventional Learningcited by 51
  4. 2023Benchmarking Large Language Model Capabilities for Conditional Generationcited by 29
  5. 2022Active Surrogate Estimators: An Active Learning Approach to Label-Efficient Model Evaluationcited by 26
  6. 2024Label-Efficient Model Selection for Text Generationcited by 19
  7. 2020Active Bayesian Assessment for Black-Box Classifierscited by 17
  8. 2024AcTracer: Active Testing of Large Language Model via Multi-Stage Samplingcited by 17
  9. 2018Active Testing: An Efficient and Robust Framework for Estimating Accuracycited by 15
  10. 2024Evaluating AI Evaluation: Perils and Prospectscited by 11
  11. 2016Classifier Risk Estimation under Limited Labeling Resourcescited by 7
  12. 2021Sample Efficient Model Evaluationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 12.

13 papers.

Sort:

By area:

  1. 2026Active Testing of Large Language Models via Approximate Neyman Allocationcited by 5
  2. 2026Adaptive auditing of AI systems with anytime-valid guaranteescited by 3
  3. 2026Rich Insights from Cheap Signals: Efficient Evaluations via Tensor Factorizationcited by 2
  4. 2026Generative Active Testing: Efficient LLM Evaluation via Proxy Task Adaptationcited by 2
  5. 2026ProEval: Proactive Failure Discovery and Efficient Performance Estimation for Generative AI Evaluationcited by 2
  6. 2026Large Language Model Selection with Limited Annotationscited by 1
  7. 2026Prediction-Powered Active Testingcited by 1
  8. 2026Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluationcited by 1
  9. 2025Large Language Model Selection with Limited Annotationscited by 0
  10. 2026BayesAME: Bayesian Active Model Evaluationcited by 0
  11. 2026MLIP Detective: Active Failure Mode Discovery Beyond Benchmark Scores for Machine-Learning Interatomic Potentialscited by 0
  12. 2026Prioritizing Repeated LLM Evaluation for Hidden Failure Discoverycited by 0
1 more
  1. 2026Which LLM to pick? Online Active Model Selection for Large Language Modelscited by 0