# Impact of Stop Sets on Stopping Active Learning for Text Classification

arXiv: https://arxiv.org/abs/2201.05460 · HTML: https://mlsanity.com/paper/2201.05460/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 42 references that are in the map.
- First posted on arXiv 8 Jan 2022.

## Builds on (most cited first, 18 of 18)

- [A Survey of Deep Active Learning](https://mlsanity.com/paper/2009.00236/) (2020)
- [A Survey of Active Learning for Text Classification using Deep Neural Networks](https://mlsanity.com/paper/2008.07267/) (2020)
- [Active Learning: Problem Settings and Recent Developments](https://mlsanity.com/paper/2012.04225/) (2020)
- [A Method for Stopping Active Learning Based on Stabilizing Predictions and the Need for User-Adjustable Stopping](https://mlsanity.com/paper/1409.5165/) (2014)
- [Autonomy and Reliability of Continuous Active Learning for Technology-Assisted Review](https://mlsanity.com/paper/1504.06868/) (2015)
- [Goldilocks: Just-Right Tuning of BERT for Technology-Assisted Review](https://mlsanity.com/paper/2105.01044/) (2021)
- [Bucking the Trend: Large-Scale Cost-Focused Active Learning for Statistical Machine Translation](https://mlsanity.com/paper/1410.5877/) (2014)
- [Hitting the Target: Stopping Active Learning at the Cost-Based Optimum](https://mlsanity.com/paper/2110.03802/) (2021)
- [Stopping Active Learning based on Predicted Change of F Measure for Text Classification](https://mlsanity.com/paper/1901.09118/) (2019)
- [Discovery of Self-Assembling π-Conjugated Peptides by Active Learning-Directed Coarse-Grained Molecular Simulation](https://mlsanity.com/paper/2002.01563/) (2020)
- [Active2 Learning: Actively reducing redundancies in Active Learning methods for Sequence Tagging and Machine Translation](https://mlsanity.com/paper/2103.06490/) (2021)
- [Taking into Account the Differences between Actively and Passively Acquired Data: The Case of Active Learning with Support Vector Machines for Imbalanced Datasets](https://mlsanity.com/paper/1409.4835/) (2014)
- [Analysis of Stopping Active Learning based on Stabilizing Predictions](https://mlsanity.com/paper/1504.06329/) (2015)
- [On Minimizing Cost in Legal Document Review Workflows](https://mlsanity.com/paper/2106.09866/) (2021)
- [The Use of Unlabeled Data versus Labeled Data for Stopping Active Learning for Text Classification](https://mlsanity.com/paper/1901.09126/) (2019)
- [Support Vector Machine Active Learning Algorithms with Query-by-Committee versus Closest-to-Hyperplane Selection](https://mlsanity.com/paper/1801.07875/) (2018)
- [Impact of Batch Size on Stopping Active Learning for Text Classification](https://mlsanity.com/paper/1801.07887/) (2018)
- [Cost-Quality Adaptive Active Learning for Chinese Clinical Named Entity Recognition](https://mlsanity.com/paper/2008.12548/) (2020)

## Built on by (most cited first, 2 of 2)

- [A Survey of Active Learning for Natural Language Processing](https://mlsanity.com/paper/2210.10109/) (2022)
- [On the Limitations of Simulating Active Learning](https://mlsanity.com/paper/2305.13342/) (2023)

## Abstract (arXiv)

Active learning is an increasingly important branch of machine learning and a powerful technique for natural language processing. The main advantage of active learning is its potential to reduce the amount of labeled data needed to learn high-performing models. A vital aspect of an effective active learning algorithm is the determination of when to stop obtaining additional labeled data. Several leading state-of-the-art stopping methods use a stop set to help make this decision. However, there has been relatively less attention given to the choice of stop set than to the stopping algorithms that are applied on the stop set. Different choices of stop sets can lead to significant differences in stopping method performance. We investigate the impact of different stop set choices on different stopping methods. This paper shows the choice of the stop set can have a significant impact on the performance of stopping methods and the impact is different for stability-based methods from that on confidence-based methods. Furthermore, the unbiased representative stop sets suggested by original authors of methods work better than the systematically biased stop sets used in recently published work, and stopping methods based on stabilizing predictions have stronger performance than confidence-based stopping methods when unbiased representative stop sets are used. We provide the largest quantity of experimental results on the impact of stop sets to date. The findings are important for helping to illuminate the impact of this important aspect of stopping methods that has been under-considered in recently published work and that can have a large practical impact on the performance of stopping methods for important semantic computing applications such as technology assisted review and text classification more broadly.