mlsanityData to 7 Oct 2026

Active Learning for Sequence Tagging with Deep Pre-trained Models and Bayesian Uncertainty Estimates

arXiv 2101.08133 ↗First posted 20 Jan 2021cs.CLcited by 39

Not on the map. Why?

Abstract (arXiv)

Annotating training data for sequence tagging of texts is usually very time-consuming. Recent advances in transfer learning for natural language processing in conjunction with active learning open the possibility to significantly reduce the necessary annotation budget. We are the first to thoroughly investigate this powerful combination for the sequence tagging task. We conduct an extensive empirical study of various Bayesian uncertainty estimation methods and Monte Carlo dropout options for deep pre-trained models in the active learning framework and find the best combinations for different types of models. Besides, we also demonstrate that to acquire instances during active learning, a full-size Transformer can be substituted with a distilled version, which yields better computational performance and reduces obstacles for applying deep active learning in practice.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Builds on 20 of its 47 references that are in the map. First posted on arXiv 20 Jan 2021.

Builds on

20 of 47 references.

References · 20

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2018Deep contextualized word representationscited by 2,285
  7. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  8. 2017Deep Bayesian Active Learning with Image Datacited by 812
  9. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  10. 2016Neural Architectures for Named Entity Recognitioncited by 548
  11. 2016End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRFcited by 341
  12. 2018A Survey on Deep Learning for Named Entity Recognitioncited by 281
8 more
  1. 2015Named Entity Recognition with Bidirectional LSTM-CNNscited by 201
  2. 2015A Theoretically Grounded Application of Dropout in Recurrent Neural Networkscited by 161
  3. 2017Deep Active Learning for Named Entity Recognitioncited by 108
  4. 2018Deep Bayesian Active Learning for Natural Language Processing: Results of a Large-Scale Empirical Studycited by 90
  5. 2017Learning how to Active Learn: A Deep Reinforcement Learning Approachcited by 79
  6. 2018Practical Obstacles to Deploying Active Learningcited by 68
  7. 2019'Zhores' -- Petaflops supercomputer for data-driven modeling, machine learning and artificial intelligence installed in Skolkovo Institute of Science and Technologycited by 23
  8. 2020Active Imitation Learning with Noisy Guidancecited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 9; 2023: 13; 2024: 7; 2025: 3; 2026: 4.

39 papers.

Sort:

By area:

  1. 2021Active Learning by Acquiring Contrastive Examplescited by 154
  2. 2022Quantifying Aleatoric and Epistemic Uncertainty in Machine Learning: Are Conditional Entropy and Mutual Information Appropriate Measures?cited by 124
  3. 2024Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraphcited by 105
  4. 2023Uncertainty in Natural Language Processing: Sources, Quantification, and Applicationscited by 44
  5. 2022A Survey of Active Learning for Natural Language Processingcited by 40
  6. 2022Active Learning Helps Pretrained Models Learn the Intended Taskcited by 33
  7. 2021On the Importance of Effectively Adapting Pretrained Language Models for Active Learningcited by 31
  8. 2024Label-wise Aleatoric and Epistemic Uncertainty Quantificationcited by 22
  9. 2022Medical Image Captioning via Generative Pretrained Transformerscited by 20
  10. 2023Second-Order Uncertainty Quantification: Variance-Based Measurescited by 19
  11. 2024ActiveLLM: Large Language Model-based Active Learning for Textual Few-Shot Scenarioscited by 17
  12. 2022Cold-Start Data Selection for Few-shot Language Model Fine-tuning: A Prompt-Based Uncertainty Propagation Approachcited by 16
27 more
  1. 2023FreeAL: Towards Human-Free Active Learning in the Era of Large Language Modelscited by 14
  2. 2022Towards Computationally Feasible Deep Active Learningcited by 11
  3. 2023On the Limitations of Simulating Active Learningcited by 9
  4. 2022FAMIE: A Fast Active Learning Framework for Multilingual Information Extractioncited by 8
  5. 2024To Label or Not to Label: Hybrid Active Learning for Neural Machine Translationcited by 7
  6. 2023Active Learning for Abstractive Text Summarizationcited by 6
  7. 2023Investigating Multi-source Active Learning for Natural Language Inferencecited by 6
  8. 2023Parameter-Efficient Language Model Tuning with Active Learning in Low-Resource Settingscited by 5
  9. 2024Hallucination Diversity-Aware Active Learning for Text Summarizationcited by 5
  10. 2025Reducing Aleatoric and Epistemic Uncertainty through Multi-modal Data Acquisitioncited by 5
  11. 2022Smooth Sailing: Improving Active Learning for Pre-trained Language Models with Representation Smoothness Analysiscited by 4
  12. 2023Re-weighting Tokens: A Simple and Effective Active Learning Strategy for Named Entity Recognitioncited by 4
  13. 2023On Dataset Transferability in Active Learning for Transformerscited by 3
  14. 2022YATO: Yet Another deep learning based Text analysis Open toolkitcited by 2
  15. 2023Realistic Conversational Question Answering with Answer Selection based on Calibrated Confidence and Uncertainty Measurementcited by 2
  16. 2024Scoping Review of Active Learning Strategies and their Evaluation Environments for Entity Recognition Taskscited by 2
  17. 2025Reassessing Active Learning Adoption in Contemporary NLP: A Community Surveycited by 2
  18. 2021AcTune: Uncertainty-aware Active Self-Training for Semi-Supervised Active Learning with Pretrained Language Modelscited by 1
  19. 2023Leveraging deep active learning to identify low-resource mobility functioning information in public clinical notescited by 1
  20. 2023Actively Discovering New Slots for Task-oriented Conversationcited by 0
  21. 2023Uncertainty Quantification using Generative Approachcited by 0
  22. 2024Towards Efficient Active Learning in NLP via Pretrained Representationscited by 0
  23. 2025ATGen: A Framework for Active Text Generationcited by 0
  24. 2026When Active Learning Falls Short: An Empirical Study on Chemical Reaction Extractioncited by 0
  25. 2026Active Learning with Foundation Model Priors: Efficient Learning under Class Imbalancecited by 0
  26. 2026Pre-AF 13: An Interpretable Atrial Fibrillation Risk Score Mined from Discharge Reportscited by 0
  27. 2026Loss-Based Active Learning for Neural Abstractive Summarizationcited by 0