mlsanityData to 7 Oct 2026

Active Testing: Sample-Efficient Model Evaluation

arXiv 2103.05331 ↗First posted 9 Mar 2021stat.ML cs.LGcited by 82

Not on the map. Why?

Abstract (arXiv)

We introduce a new framework for sample-efficient model evaluation that we call active testing. While approaches like active learning reduce the number of labels needed for model training, existing literature largely ignores the cost of labeling test data, typically unrealistically assuming large test sets for model evaluation. This creates a disconnect to real applications, where test labels are important and just as expensive, e.g. for optimizing hyperparameters. Active testing addresses this by carefully selecting the test points to label, ensuring model evaluation is sample-efficient. To this end, we derive theoretically-grounded and intuitive acquisition strategies that are specifically tailored to the goals of active testing, noting these are distinct to those of active learning. As actively selecting labels introduces a bias; we further show how to remove this bias while reducing the variance of the estimator at the same time. Active testing is easy to implement and can be applied to any supervised machine learning method. We demonstrate its effectiveness on models including WideResNets and Gaussian processes on datasets including Fashion-MNIST and CIFAR-100.

Key facts. Cited by 82 papers in the map (data through 7 Oct 2026). Builds on 19 of its 58 references that are in the map. First posted on arXiv 9 Mar 2021.

Builds on

19 of 58 references.

References · 19

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  5. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  6. 2016Wide Residual Networkscited by 3,057
  7. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
  8. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  9. 2014Semi-Supervised Learning with Deep Generative Modelscited by 494
  10. 2019BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learningcited by 428
  11. 2021Deep Adaptive Design: Amortizing Sequential Bayesian Experimental Designcited by 103
  12. 2021On Statistical Bias In Active Learning: How and When To Fix Itcited by 72
7 more
  1. 2018Practical Obstacles to Deploying Active Learningcited by 68
  2. 2019A Unified Stochastic Gradient Approach to Designing Bayesian-Optimal Experimentscited by 58
  3. 2019Radial Bayesian Neural Networks: Beyond Discrete Support In Large-Scale Bayesian Deep Learningcited by 37
  4. 2020Active Bayesian Assessment for Black-Box Classifierscited by 16
  5. 2018Active Testing: An Efficient and Robust Framework for Estimating Accuracycited by 15
  6. 2016Classifier Risk Estimation under Limited Labeling Resourcescited by 7
  7. 2019Automated Model Selection with Bayesian Quadraturecited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 7; 2023: 5; 2024: 14; 2025: 17; 2026: 36.

82 papers.

Sort:

By area:

  1. 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 295
  2. 2023Modern Bayesian Experimental Designcited by 121
  3. 2023Anchor Points: Benchmarking Models with Much Fewer Examplescited by 84
  4. 2022Evaluation Gaps in Machine Learning Practicecited by 48
  5. 2024Active Evaluation Acquisition for Efficient LLM Benchmarkingcited by 33
  6. 2021Causal-BALD: Deep Bayesian Active Learning of Outcomes to Infer Treatment-Effects from Observational Datacited by 27
  7. 2022Active Surrogate Estimators: An Active Learning Approach to Label-Efficient Model Evaluationcited by 26
  8. 2025How Benchmark Prediction from Fewer Data Misses the Markcited by 26
  9. 2023Navigating the Pitfalls of Active Learning Evaluation: A Systematic Framework for Meaningful Performance Assessmentcited by 25
  10. 2024Deep Bayesian Active Learning for Preference Modeling in Large Language Modelscited by 19
  11. 2024Label-Efficient Model Selection for Text Generationcited by 17
  12. 2024AcTracer: Active Testing of Large Language Model via Multi-Stage Samplingcited by 16
38 more
  1. 2024A Short Survey on Importance Weighting for Machine Learningcited by 15
  2. 2024A Framework for Efficient Model Evaluation through Stratification, Sampling, and Estimationcited by 13
  3. 2024ONEBench to Test Them All: Sample-Level Benchmarking Over Open-Ended Capabilitiescited by 13
  4. 2025Scaling Up Active Testing to Large Language Modelscited by 12
  5. 2026Calibrated Decisions at Scale: Converting Police Crash Narratives into Probabilistic Crash Variables with a System One Model (Jev)cited by 11
  6. 2024Efficient Lifelong Model Evaluation in an Era of Rapid Progresscited by 10
  7. 2026Multiple-Prediction-Powered Inferencecited by 9
  8. 2024All models are wrong, some are useful: Model Selection with Limited Labelscited by 8
  9. 2025When to retrain a machine learning modelcited by 8
  10. 2025How Reliable is Language Model Micro-Benchmarking?cited by 8
  11. 2025Rethinking LLM Evaluation: Can We Evaluate LLMs with 200x Less Data?cited by 8
  12. 2023DISCount: Counting in Large Image Collections with Detector-Based Importance Samplingcited by 7
  13. 2025Efficient Evaluation of Multi-Task Robot Policies With Active Experiment Selectioncited by 7
  14. 2025Consensus-Driven Active Model Selectioncited by 7
  15. 2026Agentic Business Process Management: A Research Manifestocited by 7
  16. 2022Predicting on the Edge: Identifying Where a Larger Model Does Bettercited by 6
  17. 2022Promises and Pitfalls of Threshold-based Auto-labelingcited by 6
  18. 2025Robust Sampling for Active Statistical Inferencecited by 5
  19. 2021A label-efficient two-sample testcited by 4
  20. 2025Evaluating multiple models using labeled and unlabeled datacited by 4
  21. 2025Active Measurement: Efficient Estimation at Scalecited by 4
  22. 2025DISCO: Diversifying Sample Condensation for Efficient Model Evaluationcited by 4
  23. 2026Active Testing of Large Language Models via Approximate Neyman Allocationcited by 4
  24. 2021Sample Efficient Model Evaluationcited by 3
  25. 2023Train 'n Trade: Foundations of Parameter Marketscited by 3
  26. 2026K-Sort Eval: Efficient Preference Evaluation for Visual Generation via Corrected VLM-as-a-Judgecited by 3
  27. 2026Task-CoEvolve: Efficient Harness Optimization via Adaptive Validation Task Selectioncited by 3
  28. 2022Window-Based Distribution Shift Detection for Deep Neural Networkscited by 2
  29. 2024Randomized Confidence Bounds for Stochastic Partial Monitoringcited by 2
  30. 2024Direct Acquisition Optimization for Low-Budget Active Learningcited by 2
  31. 2024Exploring Human-in-the-Loop Test-Time Adaptation by Synergizing Active Learning and Model Selectioncited by 2
  32. 2025FraPPE: Fast and Efficient Preference-based Pure Explorationcited by 2
  33. 2026Interpreting and Controlling Model Behavior via Constitutions for Atomic Concept Editscited by 2
  34. 2026Select, Label, Evaluate: Active Testing in NLPcited by 2
  35. 2026ProEval: Proactive Failure Discovery and Efficient Performance Estimation for Generative AI Evaluationcited by 2
  36. 2026Pay Only for Disagreement: Certified No-Regression Verdicts for Model Updates with Matching Label-Complexity Boundscited by 2
  37. 2025Incremental Uncertainty-aware Performance Monitoring with Active Labeling Interventioncited by 1
  38. 2025MEDAL: A Framework for Benchmarking LLMs as Multilingual Open-Domain Dialogue Evaluatorscited by 1

50 most cited of 82 All 82 (JSON)