# Deep Bayesian Active Learning for Natural Language Processing: Results of a Large-Scale Empirical Study

arXiv: https://arxiv.org/abs/1808.05697 · HTML: https://mlsanity.com/paper/1808.05697/

## Key facts

- Cited by 90 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Aug 2018.

## Built on by (most cited first, 30 of 90)

- [A Holistic Approach to Undesired Content Detection in the Real World](https://mlsanity.com/paper/2208.03274/) (2022)
- [Active Learning by Acquiring Contrastive Examples](https://mlsanity.com/paper/2109.03764/) (2021)
- [Modern Bayesian Experimental Design](https://mlsanity.com/paper/2302.14545/) (2023)
- [Plex: Towards Reliability using Pretrained Large Model Extensions](https://mlsanity.com/paper/2207.07411/) (2022)
- [On Statistical Bias In Active Learning: How and When To Fix It](https://mlsanity.com/paper/2101.11665/) (2021)
- [Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answering](https://mlsanity.com/paper/2107.02331/) (2021)
- [Prediction-Oriented Bayesian Active Learning](https://mlsanity.com/paper/2304.08151/) (2023)
- [Uncertainty in Natural Language Generation: From Theory to Applications](https://mlsanity.com/paper/2307.15703/) (2023)
- [Active Instruction Tuning: Improving Cross-Task Generalization by Training on Prompt Sensitive Tasks](https://mlsanity.com/paper/2311.00288/) (2023)
- [AfroLM: A Self-Active Learning-based Multilingual Pretrained Language Model for 23 African Languages](https://mlsanity.com/paper/2211.03263/) (2022)
- [Uncertainty in Natural Language Processing: Sources, Quantification, and Applications](https://mlsanity.com/paper/2306.04459/) (2023)
- [A Survey of Active Learning for Natural Language Processing](https://mlsanity.com/paper/2210.10109/) (2022)
- [Active Learning for Sequence Tagging with Deep Pre-trained Models and Bayesian Uncertainty Estimates](https://mlsanity.com/paper/2101.08133/) (2021)
- [On the Importance of Effectively Adapting Pretrained Language Models for Active Learning](https://mlsanity.com/paper/2104.08320/) (2021)
- [Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Models](https://mlsanity.com/paper/2408.02085/) (2024)
- [Cartography Active Learning](https://mlsanity.com/paper/2109.04282/) (2021)
- [The Efficiency Spectrum of Large Language Models: An Algorithmic Survey](https://mlsanity.com/paper/2312.00678/) (2023)
- [Exploring Predictive Uncertainty and Calibration in NLP: A Study on the Impact of Method & Data Scarcity](https://mlsanity.com/paper/2210.15452/) (2022)
- [Uncertainty-Aware Reliable Text Classification](https://mlsanity.com/paper/2107.07114/) (2021)
- [Diversity Enhanced Active Learning with Strictly Proper Scoring Rules](https://mlsanity.com/paper/2110.14171/) (2021)
- [BayesFormer: Transformer with Uncertainty Estimation](https://mlsanity.com/paper/2206.00826/) (2022)
- [Deep Bayesian Active Learning for Preference Modeling in Large Language Models](https://mlsanity.com/paper/2406.10023/) (2024)
- [Deep Learning for Bayesian Optimization of Scientific Problems with High-Dimensional Structure](https://mlsanity.com/paper/2104.11667/) (2021)
- [CAVEN: An Embodied Conversational Agent for Efficient Audio-Visual Navigation in Noisy Environments](https://mlsanity.com/paper/2306.04047/) (2023)
- [A Framework for Efficient Model Evaluation through Stratification, Sampling, and Estimation](https://mlsanity.com/paper/2406.07320/) (2024)
- [Make The Most of Prior Data: A Solution for Interactive Text Summarization with Preference Feedback](https://mlsanity.com/paper/2204.05512/) (2022)
- [Multi-class Text Classification using BERT-based Active Learning](https://mlsanity.com/paper/2104.14289/) (2021)
- [Should We Trust This Summary? Bayesian Abstractive Summarization to The Rescue](https://mlsanity.com/paper/2105.10155/) (2021)
- [Active Learning Over Multiple Domains in Natural Language Tasks](https://mlsanity.com/paper/2202.00254/) (2022)
- [Towards Computationally Feasible Deep Active Learning](https://mlsanity.com/paper/2205.03598/) (2022)

## Abstract (arXiv)

Several recent papers investigate Active Learning (AL) for mitigating the data dependence of deep learning for natural language processing. However, the applicability of AL to real-world problems remains an open question. While in supervised learning, practitioners can try many different methods, evaluating each against a validation set before selecting a model, AL affords no such luxury. Over the course of one AL run, an agent annotates its dataset exhausting its labeling budget. Thus, given a new task, an active learner has no opportunity to compare models and acquisition functions. This paper provides a large scale empirical study of deep active learning, addressing multiple tasks and, for each, multiple datasets, multiple models, and a full suite of acquisition functions. We find that across all settings, Bayesian active learning by disagreement, using uncertainty estimates provided either by Dropout or Bayes-by Backprop significantly improves over i.i.d. baselines and usually outperforms classic uncertainty sampling.