# Active Testing: An Efficient and Robust Framework for Estimating Accuracy

arXiv: https://arxiv.org/abs/1807.00493 · HTML: https://mlsanity.com/paper/1807.00493/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jul 2018.

## Built on by (most cited first, 15 of 15)

- [Active Testing: Sample-Efficient Model Evaluation](https://mlsanity.com/paper/2103.05331/) (2021)
- [On Statistical Bias In Active Learning: How and When To Fix It](https://mlsanity.com/paper/2101.11665/) (2021)
- [Mandoline: Model Evaluation under Distribution Shift](https://mlsanity.com/paper/2107.00643/) (2021)
- [Scaling Up Active Testing to Large Language Models](https://mlsanity.com/paper/2508.09093/) (2025)
- [Consensus-Driven Active Model Selection](https://mlsanity.com/paper/2507.23771/) (2025)
- [Low-Shot Validation: Active Importance Sampling for Estimating Classifier Performance on Rare Categories](https://mlsanity.com/paper/2109.05720/) (2021)
- [DISCount: Counting in Large Image Collections with Detector-Based Importance Sampling](https://mlsanity.com/paper/2306.03151/) (2023)
- [Active Measurement: Efficient Estimation at Scale](https://mlsanity.com/paper/2507.01372/) (2025)
- [Sample Efficient Model Evaluation](https://mlsanity.com/paper/2109.12043/) (2021)
- [Select, Label, Evaluate: Active Testing in NLP](https://mlsanity.com/paper/2603.21840/) (2026)
- [ProEval: Proactive Failure Discovery and Efficient Performance Estimation for Generative AI Evaluation](https://mlsanity.com/paper/2604.23099/) (2026)
- [How important is Recall for Measuring Retrieval Quality?](https://mlsanity.com/paper/2512.20854/) (2025)
- [Fault-Tolerant Evaluation for Sample-Efficient Model Performance Estimators](https://mlsanity.com/paper/2602.07226/) (2026)
- [MLIP Detective: Active Failure Mode Discovery Beyond Benchmark Scores for Machine-Learning Interatomic Potentials](https://mlsanity.com/paper/2609.08399/) (2026)
- [Tail-Influence Sampling for CVaR Policy Evaluation](https://mlsanity.com/paper/2609.38096/) (2026)

## Abstract (arXiv)

Much recent work on visual recognition aims to scale up learning to massive, noisily-annotated datasets. We address the problem of scaling- up the evaluation of such models to large-scale datasets with noisy labels. Current protocols for doing so require a human user to either vet (re-annotate) a small fraction of the test set and ignore the rest, or else correct errors in annotation as they are found through manual inspection of results. In this work, we re-formulate the problem as one of active testing, and examine strategies for efficiently querying a user so as to obtain an accu- rate performance estimate with minimal vetting. We demonstrate the effectiveness of our proposed active testing framework on estimating two performance metrics, Precision@K and mean Average Precision, for two popular computer vision tasks, multi-label classification and instance segmentation. We further show that our approach is able to save significant human annotation effort and is more robust than alternative evaluation protocols.