mlsanityData to 7 Oct 2026

Towards Computationally Feasible Deep Active Learning

arXiv 2205.03598 ↗First posted 7 May 2022cs.CL cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

Active learning (AL) is a prominent technique for reducing the annotation effort required for training machine learning models. Deep learning offers a solution for several essential obstacles to deploying AL in practice but introduces many others. One of such problems is the excessive computational resources required to train an acquisition model and estimate its uncertainty on instances in the unlabeled pool. We propose two techniques that tackle this issue for text classification and tagging tasks, offering a substantial reduction of AL iteration duration and the computational overhead introduced by deep acquisition models in AL. We also demonstrate that our algorithm that leverages pseudo-labeling and distilled models overcomes one of the essential obstacles revealed previously in the literature. Namely, it was shown that due to differences between an acquisition model used to select instances during AL and a successor model trained on the labeled data, the benefits of AL can diminish. We show that our algorithm, despite using a smaller and faster acquisition model, is capable of training a more expressive successor model with higher performance.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 24 of its 49 references that are in the map. First posted on arXiv 7 May 2022.

Builds on

24 of 49 references.

References · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  7. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  8. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  9. 2018Deep contextualized word representationscited by 2,285
  10. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,472
  11. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 565
  12. 2016End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRFcited by 341
12 more
  1. 2020Cold-start Active Learning through Self-supervised Language Modelingcited by 128
  2. 2017Deep Active Learning for Named Entity Recognitioncited by 108
  3. 2018Deep Bayesian Active Learning for Natural Language Processing: Results of a Large-Scale Empirical Studycited by 90
  4. 2017Learning how to Active Learn: A Deep Reinforcement Learning Approachcited by 79
  5. 2021Contrastive Out-of-Distribution Detection for Pretrained Transformerscited by 70
  6. 2018Practical Obstacles to Deploying Active Learningcited by 68
  7. 2021Revisiting Mahalanobis Distance for Transformer-Based Out-of-Domain Detectioncited by 67
  8. 2021Active Learning for Sequence Tagging with Deep Pre-trained Models and Bayesian Uncertainty Estimatescited by 39
  9. 2019Sampling Bias in Deep Active Classification: An Empirical Studycited by 36
  10. 2020Active Imitation Learning with Noisy Guidancecited by 15
  11. 2017Do Convolutional Networks need to be Deep for Text Classification ?cited by 11
  12. 2019Minimal Variance Sampling in Stochastic Gradient Boostingcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 4; 2024: 3; 2025: 0; 2026: 1.

11 papers.

  1. 2024Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraphcited by 105
  2. 2022A Survey of Active Learning for Natural Language Processingcited by 40
  3. 2024ActiveLLM: Large Language Model-based Active Learning for Textual Few-Shot Scenarioscited by 17
  4. 2022Automatic Document Selection for Efficient Encoder Pretrainingcited by 11
  5. 2023Active Learning for Abstractive Text Summarizationcited by 6
  6. 2024Language Model-Driven Data Pruning Enables Efficient Active Learningcited by 6
  7. 2023Interactive Multi-fidelity Learning for Cost-effective Adaptation of Language Model with Sparse Human Supervisioncited by 4
  8. 2022Towards Rapid Prototyping and Comparability in Active Learning for Deep Object Detectioncited by 3
  9. 2023On Dataset Transferability in Active Learning for Transformerscited by 3
  10. 2023Transfer and Active Learning for Dissonance Detection: Addressing the Rare-Class Challengecited by 1
  11. 2026Pre-AF 13: An Interpretable Atrial Fibrillation Risk Score Mined from Discharge Reportscited by 0