# Diversity Enhanced Active Learning with Strictly Proper Scoring Rules

arXiv: https://arxiv.org/abs/2110.14171 · HTML: https://mlsanity.com/paper/2110.14171/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 46 references that are in the map.
- First posted on arXiv 27 Oct 2021.

## Builds on (most cited first, 26 of 26)

- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Deep Bayesian Active Learning with Image Data](https://mlsanity.com/paper/1703.02910/) (2017)
- [Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds](https://mlsanity.com/paper/1906.03671/) (2019)
- [A Survey of Deep Active Learning](https://mlsanity.com/paper/2009.00236/) (2020)
- [Bayesian Deep Learning and a Probabilistic Perspective of Generalization](https://mlsanity.com/paper/2002.08791/) (2020)
- [BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learning](https://mlsanity.com/paper/1906.08158/) (2019)
- [Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping](https://mlsanity.com/paper/2002.06305/) (2020)
- [Variational Adversarial Active Learning](https://mlsanity.com/paper/1904.00370/) (2019)
- [Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning](https://mlsanity.com/paper/2002.06470/) (2020)
- [Cold-start Active Learning through Self-supervised Language Modeling](https://mlsanity.com/paper/2010.09535/) (2020)
- [PubMed 200k RCT: a Dataset for Sequential Sentence Classification in Medical Abstracts](https://mlsanity.com/paper/1710.06071/) (2017)
- [Deep Active Learning: Unified and Principled Method for Query and Training](https://mlsanity.com/paper/1911.09162/) (2019)
- [Deep Bayesian Active Learning for Natural Language Processing: Results of a Large-Scale Empirical Study](https://mlsanity.com/paper/1808.05697/) (2018)
- [Diverse mini-batch Active Learning](https://mlsanity.com/paper/1901.05954/) (2019)
- [Simple, Robust and Optimal Ranking from Pairwise Comparisons](https://mlsanity.com/paper/1512.08949/) (2015)
- [Bayesian Batch Active Learning as Sparse Subset Approximation](https://mlsanity.com/paper/1908.02144/) (2019)
- [A Survey of Active Learning for Text Classification using Deep Neural Networks](https://mlsanity.com/paper/2008.07267/) (2020)
- [Active Learning Using Uncertainty Information](https://mlsanity.com/paper/1702.08540/) (2017)
- [Cost-Accuracy Aware Adaptive Labeling for Active Learning](https://mlsanity.com/paper/2105.11418/) (2021)
- [Capsule Networks against Medical Imaging Data Challenges](https://mlsanity.com/paper/1807.07559/) (2018)
- [Diversity-Aware Batch Active Learning for Dependency Parsing](https://mlsanity.com/paper/2104.13936/) (2021)
- [Investigating the Effectiveness of Representations Based on Pretrained Transformer-based Language Models in Active Learning for Labelling Text Datasets](https://mlsanity.com/paper/2004.13138/) (2020)

## Built on by (most cited first, 22 of 22)

- [Prediction-Oriented Bayesian Active Learning](https://mlsanity.com/paper/2304.08151/) (2023)
- [A Survey on Deep Active Learning: Recent Advances and New Frontiers](https://mlsanity.com/paper/2405.00334/) (2024)
- [A Survey of Active Learning for Natural Language Processing](https://mlsanity.com/paper/2210.10109/) (2022)
- [ActiveGLAE: A Benchmark for Deep Active Learning with Transformers](https://mlsanity.com/paper/2306.10087/) (2023)
- [Improved Algorithms for Neural Active Learning](https://mlsanity.com/paper/2210.00423/) (2022)
- [Neural Active Learning Beyond Bandits](https://mlsanity.com/paper/2404.12522/) (2024)
- [Does Informativeness Matter? Active Learning for Educational Dialogue Act Classification](https://mlsanity.com/paper/2304.05578/) (2023)
- [Advancing Deep Active Learning & Data Subset Selection: Unifying Principles with Information-Theory Intuitions](https://mlsanity.com/paper/2401.04305/) (2024)
- [AUC Maximization for Low-Resource Named Entity Recognition](https://mlsanity.com/paper/2212.04800/) (2022)
- [Bayesian Estimate of Mean Proper Scores for Diversity-Enhanced Active Learning](https://mlsanity.com/paper/2312.10116/) (2023)
- [Robust Educational Dialogue Act Classifiers with Low-Resource and Imbalanced Datasets](https://mlsanity.com/paper/2304.07499/) (2023)
- [Re-weighting Tokens: A Simple and Effective Active Learning Strategy for Named Entity Recognition](https://mlsanity.com/paper/2311.00906/) (2023)
- [Improving Active Learning with a Bayesian Representation of Epistemic Uncertainty](https://mlsanity.com/paper/2412.08225/) (2024)
- [Scalable Batch-Mode Deep Bayesian Active Learning via Equivalence Class Annealing](https://mlsanity.com/paper/2112.13737/) (2021)
- [Efficient Bayesian Updates for Deep Active Learning via Laplace Approximations](https://mlsanity.com/paper/2210.06112/) (2022)
- [XAL: EXplainable Active Learning Makes Classifiers Better Low-resource Learners](https://mlsanity.com/paper/2310.05502/) (2023)
- [Exploring Active Learning in Meta-Learning: Enhancing Context Set Labeling](https://mlsanity.com/paper/2311.02879/) (2023)
- [Harnessing the Power of Beta Scoring in Deep Active Learning for Multi-Label Text Classification](https://mlsanity.com/paper/2401.07395/) (2024)
- [Tyger: Task-Type-Generic Active Learning for Molecular Property Prediction](https://mlsanity.com/paper/2205.11279/) (2022)
- [PCDAL: A Perturbation Consistency-Driven Active Learning Approach for Medical Image Segmentation and Classification](https://mlsanity.com/paper/2306.16918/) (2023)
- [Towards Efficient Active Learning in NLP via Pretrained Representations](https://mlsanity.com/paper/2402.15613/) (2024)
- [Uncertainty-Guided Edge Learning for Deep Image Regression in Remote Sensing](https://mlsanity.com/paper/2605.05590/) (2026)

## Abstract (arXiv)

We study acquisition functions for active learning (AL) for text classification. The Expected Loss Reduction (ELR) method focuses on a Bayesian estimate of the reduction in classification error, recently updated with Mean Objective Cost of Uncertainty (MOCU). We convert the ELR framework to estimate the increase in (strictly proper) scores like log probability or negative mean square error, which we call Bayesian Estimate of Mean Proper Scores (BEMPS). We also prove convergence results borrowing techniques used with MOCU. In order to allow better experimentation with the new acquisition functions, we develop a complementary batch AL algorithm, which encourages diversity in the vector of expected changes in scores for unlabelled data. To allow high performance text classifiers, we combine ensembling and dynamic validation set construction on pretrained language models. Extensive experimental evaluation then explores how these different acquisition functions perform. The results show that the use of mean square error and log probability with BEMPS yields robust acquisition functions, which consistently outperform the others tested.