# Active Surrogate Estimators: An Active Learning Approach to Label-Efficient Model Evaluation

arXiv: https://arxiv.org/abs/2202.06881 · HTML: https://mlsanity.com/paper/2202.06881/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 82 references that are in the map.
- First posted on arXiv 14 Feb 2022.

## Builds on (most cited first, 30 of 32)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [Deep Bayesian Active Learning with Image Data](https://mlsanity.com/paper/1703.02910/) (2017)
- [AutoML: A Survey of the State-of-the-Art](https://mlsanity.com/paper/1908.00709/) (2019)
- [Meta-Learning: A Survey](https://mlsanity.com/paper/1810.03548/) (2018)
- [Advances in Variational Inference](https://mlsanity.com/paper/1711.05597/) (2017)
- [Hyper-Parameter Optimization: A Review of Algorithms and Applications](https://mlsanity.com/paper/2003.05689/) (2020)
- [Deep Adaptive Design: Amortizing Sequential Bayesian Experimental Design](https://mlsanity.com/paper/2103.02438/) (2021)
- [Are Labels Always Necessary for Classifier Accuracy Evaluation?](https://mlsanity.com/paper/2007.02915/) (2020)
- [Active Testing: Sample-Efficient Model Evaluation](https://mlsanity.com/paper/2103.05331/) (2021)
- [On Statistical Bias In Active Learning: How and When To Fix It](https://mlsanity.com/paper/2101.11665/) (2021)
- [Practical Obstacles to Deploying Active Learning](https://mlsanity.com/paper/1807.04801/) (2018)
- [What Does Rotation Prediction Tell Us about Classifier Accuracy under Varying Testing Environments?](https://mlsanity.com/paper/2106.05961/) (2021)
- [Computing the Testing Error without a Testing Set](https://mlsanity.com/paper/2005.00450/) (2020)
- [Radial Bayesian Neural Networks: Beyond Discrete Support In Large-Scale Bayesian Deep Learning](https://mlsanity.com/paper/1907.00865/) (2019)
- [Probabilistic Integration: A Role in Statistical Computation?](https://mlsanity.com/paper/1512.00933/) (2015)
- [Convergence Rates of Active Learning for Maximum Likelihood Estimation](https://mlsanity.com/paper/1506.02348/) (2015)
- [Convergence Guarantees for Adaptive Bayesian Quadrature Methods](https://mlsanity.com/paper/1905.10271/) (2019)
- [Beyond Marginal Uncertainty: How Accurately can Bayesian Regression Models Estimate Posterior Predictive Correlations?](https://mlsanity.com/paper/2011.03178/) (2020)
- [Convergence of Uncertainty Sampling for Active Learning](https://mlsanity.com/paper/2110.15784/) (2021)
- [Active Bayesian Assessment for Black-Box Classifiers](https://mlsanity.com/paper/2002.06532/) (2020)
- [Targeted Active Learning for Bayesian Decision-Making](https://mlsanity.com/paper/2106.04193/) (2021)
- [Label-Free Model Evaluation with Semi-Structured Dataset Representations](https://mlsanity.com/paper/2112.00694/) (2021)
- [Active Regression by Stratification](https://mlsanity.com/paper/1410.5920/) (2014)

## Built on by (most cited first, 26 of 26)

- [Modern Bayesian Experimental Design](https://mlsanity.com/paper/2302.14545/) (2023)
- [Anchor Points: Benchmarking Models with Much Fewer Examples](https://mlsanity.com/paper/2309.08638/) (2023)
- [How Benchmark Prediction from Fewer Data Misses the Mark](https://mlsanity.com/paper/2506.07673/) (2025)
- [AcTracer: Active Testing of Large Language Model via Multi-Stage Sampling](https://mlsanity.com/paper/2408.03573/) (2024)
- [A Framework for Efficient Model Evaluation through Stratification, Sampling, and Estimation](https://mlsanity.com/paper/2406.07320/) (2024)
- [Scaling Up Active Testing to Large Language Models](https://mlsanity.com/paper/2508.09093/) (2025)
- [FlashEval: Towards Fast and Accurate Evaluation of Text-to-image Diffusion Generative Models](https://mlsanity.com/paper/2403.16379/) (2024)
- [Efficient Lifelong Model Evaluation in an Era of Rapid Progress](https://mlsanity.com/paper/2402.19472/) (2024)
- [Consensus-Driven Active Model Selection](https://mlsanity.com/paper/2507.23771/) (2025)
- [DISCO: Diversifying Sample Condensation for Efficient Model Evaluation](https://mlsanity.com/paper/2510.07959/) (2025)
- [Active Testing of Large Language Models via Approximate Neyman Allocation](https://mlsanity.com/paper/2605.10075/) (2026)
- [Evaluating multiple models using labeled and unlabeled data](https://mlsanity.com/paper/2501.11866/) (2025)
- [Learning to Rank for Active Learning via Multi-Task Bilevel Optimization](https://mlsanity.com/paper/2310.17044/) (2023)
- [Exploring Human-in-the-Loop Test-Time Adaptation by Synergizing Active Learning and Model Selection](https://mlsanity.com/paper/2405.18911/) (2024)
- [Interpreting and Controlling Model Behavior via Constitutions for Atomic Concept Edits](https://mlsanity.com/paper/2602.00092/) (2026)
- [Generative Active Testing: Efficient LLM Evaluation via Proxy Task Adaptation](https://mlsanity.com/paper/2603.19264/) (2026)
- [ProEval: Proactive Failure Discovery and Efficient Performance Estimation for Generative AI Evaluation](https://mlsanity.com/paper/2604.23099/) (2026)
- [Incremental Uncertainty-aware Performance Monitoring with Active Labeling Intervention](https://mlsanity.com/paper/2505.07023/) (2025)
- [DeltaSelect: Affordable A/B Testing for Coding Agents](https://mlsanity.com/paper/2609.19607/) (2026)
- [Novel Deep Neural Network Classifier Characterization Metrics with Applications to Dataless Evaluation](https://mlsanity.com/paper/2407.13000/) (2024)
- [Using Synthetic Data to estimate the True Error is theoretically and practically doable](https://mlsanity.com/paper/2511.00964/) (2025)
- [Label-Efficient Monitoring of Classification Models via Stratified Importance Sampling](https://mlsanity.com/paper/2601.22326/) (2026)
- [ShaplEIG: Bayesian Experimental Design for Shapley Value Estimation](https://mlsanity.com/paper/2606.02247/) (2026)
- [MLIP Detective: Active Failure Mode Discovery Beyond Benchmark Scores for Machine-Learning Interatomic Potentials](https://mlsanity.com/paper/2609.08399/) (2026)
- [Targeted Review for AI-Assisted Biodiversity Surveys: Active Continuous-Score Occupancy Modeling](https://mlsanity.com/paper/2609.25657/) (2026)
- [Speculative Evaluation of Stochastic LLMs](https://mlsanity.com/paper/2609.28560/) (2026)

## Abstract (arXiv)

We propose Active Surrogate Estimators (ASEs), a new method for label-efficient model evaluation. Evaluating model performance is a challenging and important problem when labels are expensive. ASEs address this active testing problem using a surrogate-based estimation approach that interpolates the errors of points with unknown labels, rather than forming a Monte Carlo estimator. ASEs actively learn the underlying surrogate, and we propose a novel acquisition strategy, XWED, that tailors this learning to the final estimation task. We find that ASEs offer greater label-efficiency than the current state-of-the-art when applied to challenging model evaluation problems for deep neural networks.