mlsanityData to 14 Oct 2026

Smooth Sailing: Improving Active Learning for Pre-trained Language Models with Representation Smoothness Analysis

arXiv 2212.11680 ↗First posted 20 Dec 2022cs.LG cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Developed to alleviate prohibitive labeling costs, active learning (AL) methods aim to reduce label complexity in supervised learning. While recent work has demonstrated the benefit of using AL in combination with large pre-trained language models (PLMs), it has often overlooked the practical challenges that hinder the effectiveness of AL. We address these challenges by leveraging representation smoothness analysis to ensure AL is feasible, that is, both effective and practicable. Firstly, we propose an early stopping technique that does not require a validation set -- often unavailable in realistic AL conditions -- and observe significant improvements over random sampling across multiple datasets and AL methods. Further, we find that task adaptation improves AL, whereas standard short fine-tuning in AL does not provide improvements over random sampling. Our work demonstrates the usefulness of representation smoothness analysis for AL and introduces an AL stopping criterion that reduces label complexity.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 34 of its 64 references that are in the map. First posted on arXiv 20 Dec 2022.

Builds on

34 of 64 references.

References · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  4. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  5. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  6. 2019Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Boundscited by 610
  7. 2016Error bounds for approximations with deep ReLU networkscited by 574
  8. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 384
  9. 2020On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselinescited by 255
  10. 2020Revisiting Few-sample BERT Fine-tuningcited by 238
  11. 2017Optimal approximation of piecewise smooth functions using deep ReLU neural networkscited by 179
  12. 2018Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionalitycited by 162
22 more
  1. 2021Active Learning by Acquiring Contrastive Examplescited by 156
  2. 2021Deep Learning Through the Lens of Example Difficultycited by 141
  3. 2020Cold-start Active Learning through Self-supervised Language Modelingcited by 128
  4. 2019Discriminative Active Learningcited by 98
  5. 2018Deep Bayesian Active Learning for Natural Language Processing: Results of a Large-Scale Empirical Studycited by 90
  6. 2021On the geometry of generalization and memorization in deep neural networkscited by 73
  7. 2018Practical Obstacles to Deploying Active Learningcited by 68
  8. 2021Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answeringcited by 62
  9. 2021Revisiting Uncertainty-based Query Strategies for Active Learning with Transformerscited by 60
  10. 2018Deep Neural Networks Learn Non-Smooth Functions Effectivelycited by 57
  11. 2019Deep learning is adaptive to intrinsic dimensionality of model smoothness in anisotropic Besov spacecited by 46
  12. 2021Active Learning for Sequence Tagging with Deep Pre-trained Models and Bayesian Uncertainty Estimatescited by 39
  13. 2019Low-resource Deep Entity Resolution with Transfer and Active Learningcited by 37
  14. 2019Sampling Bias in Deep Active Classification: An Empirical Studycited by 36
  15. 2021On the Importance of Effectively Adapting Pretrained Language Models for Active Learningcited by 32
  16. 2016Active Learning for Speech Recognition: the Power of Gradientscited by 25
  17. 2020Fine-tuning BERT for Low-Resource Natural Language Understanding via Active Learningcited by 25
  18. 2014A Method for Stopping Active Learning Based on Stabilizing Predictions and the Need for User-Adjustable Stoppingcited by 21
  19. 2022Measures of Information Reflect Memorization Patternscited by 12
  20. 2018Multi-Task Active Learning for Neural Semantic Role Labeling on Low Resource Conversational Corpuscited by 6
  21. 2021Stopping Criterion for Active Learning Based on Error Stabilitycited by 6
  22. 2017Function space analysis of deep learning representation layerscited by 2

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 1; 2025: 1.

4 papers.

  1. 2023ActiveGLAE: A Benchmark for Deep Active Learning with Transformerscited by 14
  2. 2023Parameter-Efficient Language Model Tuning with Active Learning in Low-Resource Settingscited by 5
  3. 2024STAR: Constraint LoRA with Dynamic Active Learning for Data-Efficient Fine-Tuning of Large Language Modelscited by 5
  4. 2025Active Learning Methods for Efficient Data Utilization and Model Performance Enhancementcited by 0