mlsanityData to 14 Oct 2026

Parameter-Efficient Language Model Tuning with Active Learning in Low-Resource Settings

arXiv 2305.14576 ↗First posted 23 May 2023cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

Pre-trained language models (PLMs) have ignited a surge in demand for effective fine-tuning techniques, particularly in low-resource domains and languages. Active learning (AL), a set of algorithms designed to decrease labeling costs by minimizing label complexity, has shown promise in confronting the labeling bottleneck. In parallel, adapter modules designed for parameter-efficient fine-tuning (PEFT) have demonstrated notable potential in low-resource settings. However, the interplay between AL and adapter-based PEFT remains unexplored. We present an empirical study of PEFT behavior with AL in low-resource settings for text classification tasks. Our findings affirm the superiority of PEFT over full-fine tuning (FFT) in low-resource settings and demonstrate that this advantage persists in AL setups. We further examine the properties of PEFT and FFT through the lens of forgetting dynamics and instance-level representations, where we find that PEFT yields more stable representations of early and middle layers compared to FFT. Our research underscores the synergistic potential of AL and PEFT in low-resource settings, paving the way for advancements in efficient and effective fine-tuning.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 29 of its 61 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

29 of 61 references.

References · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  6. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  7. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  8. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  9. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  10. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 678
  11. 2020AdapterHub: A Framework for Adapting Transformerscited by 444
  12. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 384
17 more
  1. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 284
  2. 2020On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselinescited by 255
  3. 2020Revisiting Few-sample BERT Fine-tuningcited by 238
  4. 2021UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuningcited by 162
  5. 2021Active Learning by Acquiring Contrastive Examplescited by 156
  6. 2021On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptationcited by 147
  7. 2021Deep Learning Through the Lens of Example Difficultycited by 141
  8. 2020Cold-start Active Learning through Self-supervised Language Modelingcited by 128
  9. 2019Discriminative Active Learningcited by 98
  10. 2023Modular Deep Learningcited by 83
  11. 2021On the geometry of generalization and memorization in deep neural networkscited by 73
  12. 2021Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answeringcited by 62
  13. 2021Revisiting Uncertainty-based Query Strategies for Active Learning with Transformerscited by 60
  14. 2021Active Learning for Sequence Tagging with Deep Pre-trained Models and Bayesian Uncertainty Estimatescited by 39
  15. 2021On the Importance of Effectively Adapting Pretrained Language Models for Active Learningcited by 32
  16. 2020Fine-tuning BERT for Low-Resource Natural Language Understanding via Active Learningcited by 25
  17. 2022Smooth Sailing: Improving Active Learning for Pre-trained Language Models with Representation Smoothness Analysiscited by 4

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 2.

5 papers.

  1. 2024Hallucination Diversity-Aware Active Learning for Text Summarizationcited by 5
  2. 2024Fine-tuning Large Language Models with Limited Data: A Survey and Practical Guidecited by 4
  3. 2024Parameter-Efficient Active Learning for Foundational modelscited by 1
  4. 2025Op-Fed: Opinion, Stance, and Monetary Policy Annotations on FOMC Transcripts Using Active Learningcited by 1
  5. 2025State Space Models for Extractive Summarization in Low Resource Scenarioscited by 0