mlsanityData to 14 Oct 2026

Diversity Enhanced Active Learning with Strictly Proper Scoring Rules

arXiv 2110.14171 ↗First posted 27 Oct 2021cs.LG cs.AI cs.CLcited by 22

Not on the map. Why?

Abstract (arXiv)

We study acquisition functions for active learning (AL) for text classification. The Expected Loss Reduction (ELR) method focuses on a Bayesian estimate of the reduction in classification error, recently updated with Mean Objective Cost of Uncertainty (MOCU). We convert the ELR framework to estimate the increase in (strictly proper) scores like log probability or negative mean square error, which we call Bayesian Estimate of Mean Proper Scores (BEMPS). We also prove convergence results borrowing techniques used with MOCU. In order to allow better experimentation with the new acquisition functions, we develop a complementary batch AL algorithm, which encourages diversity in the vector of expected changes in scores for unlabelled data. To allow high performance text classifiers, we combine ensembling and dynamic validation set construction on pretrained language models. Extensive experimental evaluation then explores how these different acquisition functions perform. The results show that the use of mean square error and log probability with BEMPS yields robust acquisition functions, which consistently outperform the others tested.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Builds on 26 of its 46 references that are in the map. First posted on arXiv 27 Oct 2021.

Builds on

26 of 46 references.

References · 26

  1. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  2. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  3. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  4. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  5. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  6. 2017Deep Bayesian Active Learning with Image Datacited by 819
  7. 2019Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Boundscited by 610
  8. 2020A Survey of Deep Active Learningcited by 554
  9. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  10. 2019BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learningcited by 432
  11. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 384
  12. 2019Variational Adversarial Active Learningcited by 284
14 more
  1. 2020Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learningcited by 210
  2. 2020Cold-start Active Learning through Self-supervised Language Modelingcited by 128
  3. 2017PubMed 200k RCT: a Dataset for Sequential Sentence Classification in Medical Abstractscited by 124
  4. 2019Deep Active Learning: Unified and Principled Method for Query and Trainingcited by 93
  5. 2018Deep Bayesian Active Learning for Natural Language Processing: Results of a Large-Scale Empirical Studycited by 90
  6. 2019Diverse mini-batch Active Learningcited by 89
  7. 2015Simple, Robust and Optimal Ranking from Pairwise Comparisonscited by 70
  8. 2019Bayesian Batch Active Learning as Sparse Subset Approximationcited by 68
  9. 2020A Survey of Active Learning for Text Classification using Deep Neural Networkscited by 50
  10. 2017Active Learning Using Uncertainty Informationcited by 28
  11. 2021Cost-Accuracy Aware Adaptive Labeling for Active Learningcited by 13
  12. 2018Capsule Networks against Medical Imaging Data Challengescited by 9
  13. 2021Diversity-Aware Batch Active Learning for Dependency Parsingcited by 9
  14. 2020Investigating the Effectiveness of Representations Based on Pretrained Transformer-based Language Models in Active Learning for Labelling Text Datasetscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 5; 2023: 9; 2024: 6; 2025: 0; 2026: 1.

22 papers.

Sort:

  1. 2023Prediction-Oriented Bayesian Active Learningcited by 62
  2. 2024A Survey on Deep Active Learning: Recent Advances and New Frontierscited by 54
  3. 2022A Survey of Active Learning for Natural Language Processingcited by 40
  4. 2023ActiveGLAE: A Benchmark for Deep Active Learning with Transformerscited by 14
  5. 2022Improved Algorithms for Neural Active Learningcited by 11
  6. 2024Neural Active Learning Beyond Banditscited by 10
  7. 2023Does Informativeness Matter? Active Learning for Educational Dialogue Act Classificationcited by 8
  8. 2024Advancing Deep Active Learning & Data Subset Selection: Unifying Principles with Information-Theory Intuitionscited by 7
  9. 2022AUC Maximization for Low-Resource Named Entity Recognitioncited by 6
  10. 2023Bayesian Estimate of Mean Proper Scores for Diversity-Enhanced Active Learningcited by 6
  11. 2023Robust Educational Dialogue Act Classifiers with Low-Resource and Imbalanced Datasetscited by 5
  12. 2023Re-weighting Tokens: A Simple and Effective Active Learning Strategy for Named Entity Recognitioncited by 5
10 more
  1. 2024Improving Active Learning with a Bayesian Representation of Epistemic Uncertaintycited by 4
  2. 2021Scalable Batch-Mode Deep Bayesian Active Learning via Equivalence Class Annealingcited by 2
  3. 2022Efficient Bayesian Updates for Deep Active Learning via Laplace Approximationscited by 2
  4. 2023XAL: EXplainable Active Learning Makes Classifiers Better Low-resource Learnerscited by 1
  5. 2023Exploring Active Learning in Meta-Learning: Enhancing Context Set Labelingcited by 1
  6. 2024Harnessing the Power of Beta Scoring in Deep Active Learning for Multi-Label Text Classificationcited by 1
  7. 2022Tyger: Task-Type-Generic Active Learning for Molecular Property Predictioncited by 0
  8. 2023PCDAL: A Perturbation Consistency-Driven Active Learning Approach for Medical Image Segmentation and Classificationcited by 0
  9. 2024Towards Efficient Active Learning in NLP via Pretrained Representationscited by 0
  10. 2026Uncertainty-Guided Edge Learning for Deep Image Regression in Remote Sensingcited by 0