# Hitting the Target: Stopping Active Learning at the Cost-Based Optimum

arXiv: https://arxiv.org/abs/2110.03802 · HTML: https://mlsanity.com/paper/2110.03802/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 46 references that are in the map.
- First posted on arXiv 7 Oct 2021.

## Builds on (most cited first, 10 of 10)

- [Discriminative Active Learning](https://mlsanity.com/paper/1907.06347/) (2019)
- [Exploring Connections Between Active Learning and Model Extraction](https://mlsanity.com/paper/1811.02054/) (2018)
- [A Benchmark and Comparison of Active Learning for Logistic Regression](https://mlsanity.com/paper/1611.08618/) (2016)
- [modAL: A modular active learning framework for Python](https://mlsanity.com/paper/1805.00979/) (2018)
- [A Method for Stopping Active Learning Based on Stabilizing Predictions and the Need for User-Adjustable Stopping](https://mlsanity.com/paper/1409.5165/) (2014)
- [Stopping criterion for active learning based on deterministic generalization bounds](https://mlsanity.com/paper/2005.07402/) (2020)
- [Active learning for structural reliability: survey, general framework and benchmark](https://mlsanity.com/paper/2106.01713/) (2021)
- [Stopping Criterion for Active Learning Based on Error Stability](https://mlsanity.com/paper/2104.01836/) (2021)
- [The Use of Unlabeled Data versus Labeled Data for Stopping Active Learning for Text Classification](https://mlsanity.com/paper/1901.09126/) (2019)
- [Deciding when to stop: Efficient stopping of active learning guided drug-target prediction](https://mlsanity.com/paper/1504.02406/) (2015)

## Built on by (most cited first, 9 of 9)

- [A Survey of Active Learning for Natural Language Processing](https://mlsanity.com/paper/2210.10109/) (2022)
- [From Selection to Generation: A Survey of LLM-based Active Learning](https://mlsanity.com/paper/2502.11767/) (2025)
- [Can LLMs Correct Themselves? A Benchmark of Self-Correction in LLMs](https://mlsanity.com/paper/2510.16062/) (2025)
- [On the Limitations of Simulating Active Learning](https://mlsanity.com/paper/2305.13342/) (2023)
- [Robust online active learning](https://mlsanity.com/paper/2302.00422/) (2023)
- [Finding Needles in the Haystack: Transductive Active Labeling in Ecology](https://mlsanity.com/paper/2606.03821/) (2026)
- [Impact of Stop Sets on Stopping Active Learning for Text Classification](https://mlsanity.com/paper/2201.05460/) (2022)
- [Autonomous Assessment of Demonstration Sufficiency via Bayesian Inverse Reinforcement Learning](https://mlsanity.com/paper/2211.15542/) (2022)
- [Survey of Active Learning Hyperparameters: Insights from a Large-Scale Experimental Grid](https://mlsanity.com/paper/2506.03817/) (2025)

## Abstract (arXiv)

Active learning allows machine learning models to be trained using fewer labels while retaining similar performance to traditional supervised learning. An active learner selects the most informative data points, requests their labels, and retrains itself. While this approach is promising, it raises the question of how to determine when the model is `good enough' without the additional labels required for traditional evaluation. Previously, different stopping criteria have been proposed aiming to identify the optimal stopping point. Yet, optimality can only be expressed as a domain-dependent trade-off between accuracy and the number of labels, and no criterion is superior in all applications. As a further complication, a comparison of criteria for a particular real-world application would require practitioners to collect additional labelled data they are aiming to avoid by using active learning in the first place. This work enables practitioners to employ active learning by providing actionable recommendations for which stopping criteria are best for a given real-world scenario. We contribute the first large-scale comparison of stopping criteria for pool-based active learning, using a cost measure to quantify the accuracy/label trade-off, public implementations of all stopping criteria we evaluate, and an open-source framework for evaluating stopping criteria. Our research enables practitioners to substantially reduce labelling costs by utilizing the stopping criterion which best suits their domain.