mlsanityData to 7 Oct 2026

On the Importance of Effectively Adapting Pretrained Language Models for Active Learning

arXiv 2104.08320 ↗First posted 16 Apr 2021cs.CLcited by 31

Not on the map. Why?

Abstract (arXiv)

Recent Active Learning (AL) approaches in Natural Language Processing (NLP) proposed using off-the-shelf pretrained language models (LMs). In this paper, we argue that these LMs are not adapted effectively to the downstream task during AL and we explore ways to address this issue. We suggest to first adapt the pretrained LM to the target task by continuing training with all the available unlabeled data and then use it for AL. We also propose a simple yet effective fine-tuning method to ensure that the adapted LM is properly trained in both low and high resource scenarios during AL. Our experiments demonstrate that our approach provides substantial data efficiency improvements compared to the standard fine-tuning approach, suggesting that a poor training strategy can be catastrophic for AL.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 24 of its 51 references that are in the map. First posted on arXiv 16 Apr 2021.

Builds on

24 of 51 references.

References · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  7. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  8. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  9. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  10. 2017Deep Bayesian Active Learning with Image Datacited by 812
  11. 2019Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Boundscited by 602
  12. 2019BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learningcited by 428
12 more
  1. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 377
  2. 2020On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselinescited by 254
  3. 2020Revisiting Few-sample BERT Fine-tuningcited by 237
  4. 2018Adversarial Active Learning for Deep Networks: a Margin Based Approachcited by 155
  5. 2020Cold-start Active Learning through Self-supervised Language Modelingcited by 128
  6. 2017Deep Active Learning for Named Entity Recognitioncited by 108
  7. 2018Deep Bayesian Active Learning for Natural Language Processing: Results of a Large-Scale Empirical Studycited by 90
  8. 2018Practical Obstacles to Deploying Active Learningcited by 68
  9. 2021Active Learning for Sequence Tagging with Deep Pre-trained Models and Bayesian Uncertainty Estimatescited by 39
  10. 2019Low-resource Deep Entity Resolution with Transfer and Active Learningcited by 37
  11. 2019Sampling Bias in Deep Active Classification: An Empirical Studycited by 36
  12. 2018Multi-Task Active Learning for Neural Semantic Role Labeling on Low Resource Conversational Corpuscited by 6

Built on by

Cited by year

Citing papers per year: 2022: 6; 2023: 12; 2024: 8; 2025: 5.

31 papers.

Sort:

By area:

  1. 2024Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AIcited by 60
  2. 2024A Survey of Uncertainty Estimation in LLMs: Theory Meets Practicecited by 54
  3. 2023Active Learning Principles for In-Context Learning with Large Language Modelscited by 53
  4. 2024A Survey on Deep Active Learning: Recent Advances and New Frontierscited by 53
  5. 2022A Survey of Active Learning for Natural Language Processingcited by 40
  6. 2023Don't Stop Pretraining? Make Prompt-based Fine-tuning Powerful Learnercited by 23
  7. 2022Fine-Tuning Language Models via Epistemic Neural Networkscited by 22
  8. 2023Stance Detection: A Practical Guide to Classifying Political Beliefs in Textcited by 20
  9. 2023Rethinking Semi-supervised Learning with Language Modelscited by 17
  10. 2022Cold-Start Data Selection for Few-shot Language Model Fine-tuning: A Prompt-Based Uncertainty Propagation Approachcited by 16
  11. 2023ActiveGLAE: A Benchmark for Deep Active Learning with Transformerscited by 14
  12. 2023D-CALM: A Dynamic Clustering-based Active Learning Approach for Mitigating Biascited by 12
19 more
  1. 2022Automatic Document Selection for Efficient Encoder Pretrainingcited by 11
  2. 2024Making Better Use of Unlabelled Data in Bayesian Active Learningcited by 10
  3. 2024Active Learning for Robust and Representative LLM Generation in Safety-Critical Scenarioscited by 10
  4. 2023On the Limitations of Simulating Active Learningcited by 9
  5. 2023Investigating Multi-source Active Learning for Natural Language Inferencecited by 6
  6. 2023Parameter-Efficient Language Model Tuning with Active Learning in Low-Resource Settingscited by 5
  7. 2024STAR: Constraint LoRA with Dynamic Active Learning for Data-Efficient Fine-Tuning of Large Language Modelscited by 5
  8. 2022Smooth Sailing: Improving Active Learning for Pre-trained Language Models with Representation Smoothness Analysiscited by 4
  9. 2024AnchorAL: Computationally Efficient Active Learning for Large and Imbalanced Datasetscited by 4
  10. 2023On Dataset Transferability in Active Learning for Transformerscited by 3
  11. 2022An Efficient Active Learning Pipeline for Legal Text Classificationcited by 2
  12. 2025Reassessing Active Learning Adoption in Contemporary NLP: A Community Surveycited by 2
  13. 2023Comprehensive Benchmarking of Entropy and Margin Based Scoring Metrics for Data Selectioncited by 1
  14. 2025No Free Lunch in Active Learning: LLM Embedding Quality Dictates Query Strategy Successcited by 1
  15. 2025Op-Fed: Opinion, Stance, and Monetary Policy Annotations on FOMC Transcripts Using Active Learningcited by 1
  16. 2023Mining of Single-Class by Active Learning for Semantic Segmentationcited by 0
  17. 2024Self-Training for Sample-Efficient Active Learning for Text Classification with Pre-Trained Language Modelscited by 0
  18. 2025Active Learning Methods for Efficient Data Utilization and Model Performance Enhancementcited by 0
  19. 2025Generalized Category Discovery in Event-Centric Contexts: Latent Pattern Mining with LLMscited by 0