mlsanityData to 14 Oct 2026

On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation

arXiv 2106.03164 ↗First posted 6 Jun 2021cs.CLcited by 147

Not on the map. Why?

Abstract (arXiv)

Adapter-based tuning has recently arisen as an alternative to fine-tuning. It works by adding light-weight adapter modules to a pretrained language model (PrLM) and only updating the parameters of adapter modules when learning on a downstream task. As such, it adds only a few trainable parameters per new task, allowing a high degree of parameter sharing. Prior studies have shown that adapter-based tuning often achieves comparable results to fine-tuning. However, existing work only focuses on the parameter-efficient aspect of adapter-based tuning while lacking further investigation on its effectiveness. In this paper, we study the latter. We first show that adapter-based tuning better mitigates forgetting issues than fine-tuning since it yields representations with less deviation from those generated by the initial PrLM. We then empirically compare the two tuning methods on several downstream NLP tasks and settings. We demonstrate that 1) adapter-based tuning outperforms fine-tuning on low-resource and cross-lingual tasks; 2) it is more robust to overfitting and less sensitive to changes in learning rates.

Key facts. Cited by 147 papers in the map (data through 14 Oct 2026). Builds on 37 of its 46 references that are in the map. First posted on arXiv 6 Jun 2021.

Builds on

37 of 46 references.

References · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  8. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  9. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  10. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  11. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  12. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
25 more
  1. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 788
  2. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  3. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 722
  4. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  5. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 445
  6. 2020AdapterHub: A Framework for Adapting Transformerscited by 444
  7. 2019How to Fine-Tune BERT for Text Classification?cited by 411
  8. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 384
  9. 2020K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapterscited by 319
  10. 2019FreeLB: Enhanced Adversarial Training for Natural Language Understandingcited by 264
  11. 2020On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselinescited by 255
  12. 2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 254
  13. 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 242
  14. 2020Revisiting Few-sample BERT Fine-tuningcited by 238
  15. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 224
  16. 2018Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Taskscited by 215
  17. 2020Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgettingcited by 170
  18. 2020Better Fine-Tuning by Reducing Representational Collapsecited by 157
  19. 2019Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Modelscited by 141
  20. 2020What Happens To BERT Embeddings During Fine-tuning?cited by 113
  21. 2019Visualizing and Understanding the Effectiveness of BERTcited by 67
  22. 2020Incorporating BERT into Parallel Sequence Decoding with Adapterscited by 39
  23. 2019Blackbox meets blackbox: Representational Similarity and Stability Analysis of Neural Language Models and Brainscited by 38
  24. 2019Correlating neural and symbolic representations of languagecited by 33
  25. 2016How Transferable are Neural Networks in NLP Applications?cited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 8; 2022: 27; 2023: 42; 2024: 39; 2025: 22; 2026: 9.

147 papers.

Sort:

By area:

  1. 2022Is GPT-3 a Good Data Annotator?cited by 208
  2. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 180
  3. 2021UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuningcited by 162
  4. 2023MotionGPT: Finetuned LLMs Are General-Purpose Motion Generatorscited by 145
  5. 2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 141
  6. 2024Let Your Graph Do the Talking: Encoding Structured Data for LLMscited by 118
  7. 2023Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speechcited by 101
  8. 2023Retrieving Multimodal Information for Augmented Generation: A Surveycited by 85
  9. 2023Modular Deep Learningcited by 83
  10. 2024LLMs are Good Sign Language Translatorscited by 77
  11. 2024LoRA-Pro: Are Low-Rank Adapters Properly Optimized?cited by 77
  12. 2022Revisiting Parameter-Efficient Tuning: Are We Really There Yet?cited by 76
38 more
  1. 2024When LLMs Meet Cybersecurity: A Systematic Literature Reviewcited by 71
  2. 2022ELLE: Efficient Lifelong Pre-training for Emerging Datacited by 70
  3. 2021Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristicscited by 67
  4. 2024MotionGPT-2: A General-Purpose Motion-Language Model for Motion Generation and Understandingcited by 59
  5. 2023ViDA: Homeostatic Visual Domain Adapter for Continual Test Time Adaptationcited by 58
  6. 2023UPop: Unified and Progressive Pruning for Compressing Vision-Language Transformerscited by 54
  7. 2025Injecting Domain-Specific Knowledge into Large Language Models: A Comprehensive Surveycited by 51
  8. 2023Revisiting the Parameter Efficiency of Adapters from the Perspective of Precision Redundancycited by 44
  9. 2022IDPG: An Instance-Dependent Prompt Generation Methodcited by 43
  10. 2022Improving Sharpness-Aware Minimization with Fisher Mask for Better Generalization on Language Modelscited by 42
  11. 2023HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Modelscited by 42
  12. 2024Revisiting Knowledge Distillation for Autoregressive Language Modelscited by 42
  13. 2023Empirical Analysis of the Strengths and Weaknesses of PEFT Techniques for LLMscited by 38
  14. 2021Robust Transfer Learning with Pretrained Language Models through Adapterscited by 35
  15. 2022Input-Tuning: Adapting Unfamiliar Inputs to Frozen Pretrained Modelscited by 34
  16. 2022Frustratingly Easy Label Projection for Cross-lingual Transfercited by 33
  17. 2023MultiLoRA: Democratizing LoRA for Better Multi-Task Learningcited by 32
  18. 2022GPT-3-driven pedagogical agents for training children's curious question-asking skillscited by 29
  19. 2022Exploring Efficient-tuning Methods in Self-supervised Speech Modelscited by 26
  20. 2022On the Domain Adaptation and Generalization of Pretrained Language Models: A Surveycited by 23
  21. 2023VLN-PETL: Parameter-Efficient Transfer Learning for Vision-and-Language Navigationcited by 23
  22. 2024An Empirical Study on Parameter-Efficient Fine-Tuning for MultiModal Large Language Modelscited by 22
  23. 2024RoseLoRA: Row and Column-wise Sparse Low-rank Adaptation of Pre-trained Language Model for Knowledge Editing and Fine-tuningcited by 21
  24. 2022Parameter-Efficient Tuning on Layer Normalization for Pre-trained Language Modelscited by 19
  25. 2023PECoP: Parameter Efficient Continual Pretraining for Action Quality Assessmentcited by 19
  26. 2022Sparse Structure Search for Parameter-Efficient Tuningcited by 17
  27. 2024The Hidden Space of Transformer Language Adapterscited by 17
  28. 2024Random Masking Finds Winning Tickets for Parameter Efficient Fine-tuningcited by 17
  29. 2022Efficient Few-Shot Fine-Tuning for Opinion Summarizationcited by 16
  30. 2023One Adapter for All Programming Languages? Adapter Tuning for Code Search and Summarizationcited by 16
  31. 2024Lessons and Insights from a Unifying Study of Parameter-Efficient Fine-Tuning (PEFT) in Visual Recognitioncited by 16
  32. 2024AFLoRA: Adaptive Freezing of Low Rank Adaptation in Parameter Efficient Fine-Tuning of Large Modelscited by 15
  33. 2024Adapting Multilingual LLMs to Low-Resource Languages with Knowledge Graphs via Adapterscited by 15
  34. 2021Unsupervised Domain Adaptation with Adaptercited by 14
  35. 2023PrivateLoRA For Efficient Privacy Preserving LLMcited by 14
  36. 2021Enhancing Multilingual Language Model with Massive Multilingual Knowledge Triplescited by 13
  37. 2022Adaptable Adapterscited by 13
  38. 2022CHAPTER: Exploiting Convolutional Neural Network Adapters for Self-supervised Speech Modelscited by 13

50 most cited of 147 All 147 (JSON)