# Practical Obstacles to Deploying Active Learning

arXiv: https://arxiv.org/abs/1807.04801 · HTML: https://mlsanity.com/paper/1807.04801/

## Key facts

- Cited by 68 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jul 2018.

## Built on by (most cited first, 30 of 68)

- [Active Learning by Acquiring Contrastive Examples](https://mlsanity.com/paper/2109.03764/) (2021)
- [Active Learning on a Budget: Opposite Strategies Suit High and Low Budgets](https://mlsanity.com/paper/2202.02794/) (2022)
- [Few-Shot Preference Learning for Human-in-the-Loop RL](https://mlsanity.com/paper/2212.03363/) (2022)
- [Active Testing: Sample-Efficient Model Evaluation](https://mlsanity.com/paper/2103.05331/) (2021)
- [Efficient Methods for Natural Language Processing: A Survey](https://mlsanity.com/paper/2209.00099/) (2022)
- [On Statistical Bias In Active Learning: How and When To Fix It](https://mlsanity.com/paper/2101.11665/) (2021)
- [Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answering](https://mlsanity.com/paper/2107.02331/) (2021)
- [Revisiting Uncertainty-based Query Strategies for Active Learning with Transformers](https://mlsanity.com/paper/2107.05687/) (2021)
- [A Survey of Active Learning for Natural Language Processing](https://mlsanity.com/paper/2210.10109/) (2022)
- [Active Learning for Sequence Tagging with Deep Pre-trained Models and Bayesian Uncertainty Estimates](https://mlsanity.com/paper/2101.08133/) (2021)
- [Active Learning Helps Pretrained Models Learn the Intended Task](https://mlsanity.com/paper/2204.08491/) (2022)
- [On the Importance of Effectively Adapting Pretrained Language Models for Active Learning](https://mlsanity.com/paper/2104.08320/) (2021)
- [On the Efficacy of Adversarial Data Collection for Question Answering: Results from a Large-Scale Randomized Study](https://mlsanity.com/paper/2106.00872/) (2021)
- [Cartography Active Learning](https://mlsanity.com/paper/2109.04282/) (2021)
- [Active Surrogate Estimators: An Active Learning Approach to Label-Efficient Model Evaluation](https://mlsanity.com/paper/2202.06881/) (2022)
- [Analyzing Dynamic Adversarial Training Data in the Limit](https://mlsanity.com/paper/2110.08514/) (2021)
- [ALLSH: Active Learning Guided by Local Sensitivity and Hardness](https://mlsanity.com/paper/2205.04980/) (2022)
- [ActiveLLM: Large Language Model-based Active Learning for Textual Few-Shot Scenarios](https://mlsanity.com/paper/2405.10808/) (2024)
- [A Cross-Domain Benchmark for Active Learning](https://mlsanity.com/paper/2408.00426/) (2024)
- [Adapting Coreference Resolution Models through Active Learning](https://mlsanity.com/paper/2104.07611/) (2021)
- [Multi-class Text Classification using BERT-based Active Learning](https://mlsanity.com/paper/2104.14289/) (2021)
- [Active Learning Over Multiple Domains in Natural Language Tasks](https://mlsanity.com/paper/2202.00254/) (2022)
- [Active Few-Shot Learning with FASL](https://mlsanity.com/paper/2204.09347/) (2022)
- [Towards Computationally Feasible Deep Active Learning](https://mlsanity.com/paper/2205.03598/) (2022)
- [Clean or Annotate: How to Spend a Limited Data Collection Budget](https://mlsanity.com/paper/2110.08355/) (2021)
- [Active Learning for Robust and Representative LLM Generation in Safety-Critical Scenarios](https://mlsanity.com/paper/2410.11114/) (2024)
- [On the Limitations of Simulating Active Learning](https://mlsanity.com/paper/2305.13342/) (2023)
- [Annotation Curricula to Implicitly Train Non-Expert Annotators](https://mlsanity.com/paper/2106.02382/) (2021)
- [FAMIE: A Fast Active Learning Framework for Multilingual Information Extraction](https://mlsanity.com/paper/2202.08316/) (2022)
- [Structurally Diverse Sampling for Sample-Efficient Training and Comprehensive Evaluation](https://mlsanity.com/paper/2203.08445/) (2022)

## Abstract (arXiv)

Active learning (AL) is a widely-used training strategy for maximizing predictive performance subject to a fixed annotation budget. In AL one iteratively selects training examples for annotation, often those for which the current model is most uncertain (by some measure). The hope is that active sampling leads to better performance than would be achieved under independent and identically distributed (i.i.d.) random samples. While AL has shown promise in retrospective evaluations, these studies often ignore practical obstacles to its use. In this paper we show that while AL may provide benefits when used with specific models and for particular domains, the benefits of current approaches do not generalize reliably across models and tasks. This is problematic because in practice one does not have the opportunity to explore and compare alternative AL strategies. Moreover, AL couples the training dataset with the model used to guide its acquisition. We find that subsequently training a successor model with an actively-acquired dataset does not consistently outperform training on i.i.d. sampled data. Our findings raise the question of whether the downsides inherent to AL are worth the modest and inconsistent performance gains it tends to afford.