mlsanityData to 7 Oct 2026

Towards a Unified View of Parameter-Efficient Transfer Learning

arXiv 2110.04366 ↗First posted 8 Oct 2021cs.CL cs.LGcited by 863

Not on the map. Why?

Cited by 38 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Fine-tuning large pre-trained language models on downstream tasks has become the de-facto learning paradigm in NLP. However, conventional approaches fine-tune all the parameters of the pre-trained model, which becomes prohibitive as the model size and the number of tasks grow. Recent work has proposed a variety of parameter-efficient transfer learning methods that only fine-tune a small number of (extra) parameters to attain strong performance. While effective, the critical ingredients for success and the connections among the various methods are poorly understood. In this paper, we break down the design of state-of-the-art parameter-efficient transfer learning methods and present a unified framework that establishes connections between them. Specifically, we re-frame them as modifications to specific hidden states in pre-trained models, and define a set of design dimensions along which different methods vary, such as the function to compute the modification and the position to apply the modification. Through comprehensive empirical studies across machine translation, text summarization, language understanding, and text classification benchmarks, we utilize the unified view to identify important design choices in previous methods. Furthermore, our unified framework enables the transfer of design elements across different approaches, and as a result we are able to instantiate new parameter-efficient fine-tuning methods that tune less parameters than previous methods while being more effective, achieving comparable results to fine-tuning all parameters on all four tasks.

Key facts. Cited by 863 papers in the map (data through 7 Oct 2026). Builds on 31 of its 41 references that are in the map. First posted on arXiv 8 Oct 2021.

Builds on

31 of 41 references.

References · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  9. 2016Layer Normalizationcited by 5,579
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  11. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  12. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
19 more
  1. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  2. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  4. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  5. 2018Deep contextualized word representationscited by 2,285
  6. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  7. 2020Extracting Training Data from Large Language Modelscited by 2,144
  8. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  9. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  10. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  11. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  12. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  13. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  14. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  15. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 504
  16. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  17. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  18. 2017The E2E Dataset: New Challenges For End-to-End Generationcited by 252
  19. 2019Identifying and Reducing Gender Bias in Word-Level Language Modelscited by 214

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 119; 2023: 209; 2024: 270; 2025: 171; 2026: 88.

863 papers.

Sort:

By area:

  1. 2022Visual Prompt Tuningcited by 1,674
  2. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  3. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  4. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  5. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  6. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  7. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  8. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  9. 2023A Comprehensive Overview of Large Language Modelscited by 639
  10. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 550
  11. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  12. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
38 more
  1. 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 337
  2. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  3. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 317
  4. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 296
  5. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  6. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 261
  7. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  8. 2022Black-Box Tuning for Language-Model-as-a-Servicecited by 254
  9. 2022Reasoning with Language Model Prompting: A Surveycited by 251
  10. 2023Personalize Segment Anything Model with One Shotcited by 250
  11. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 242
  12. 2023Ring-A-Bell! How Reliable are Concept Removal Methods for Diffusion Models?cited by 239
  13. 2022Making Large Language Models Better Reasoners with Step-Aware Verifiercited by 231
  14. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 229
  15. 2022LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learningcited by 228
  16. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 228
  17. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 227
  18. 2024When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Methodcited by 201
  19. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 195
  20. 2022ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learningcited by 192
  21. 2024ReFT: Representation Finetuning for Language Modelscited by 186
  22. 2023Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Needcited by 183
  23. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 179
  24. 2023ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Modelscited by 174
  25. 2023Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuningcited by 171
  26. 2023RanPAC: Random Projections and Pre-trained Models for Continual Learningcited by 162
  27. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
  28. 2024Improving LoRA in Privacy-preserving Federated Learningcited by 158
  29. 2023Composing Parameter-Efficient Modules with Arithmetic Operationscited by 155
  30. 2023QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Modelscited by 154
  31. 2023Platypus: Quick, Cheap, and Powerful Refinement of LLMscited by 150
  32. 2021OpenPrompt: An Open-source Framework for Prompt-learningcited by 148
  33. 2023AIM: Adapting Image Models for Efficient Video Action Recognitioncited by 145
  34. 2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Expertscited by 144
  35. 2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 140
  36. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
  37. 2022Neural Prompt Searchcited by 133
  38. 2022FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformercited by 133

50 most cited of 863 All 863 (JSON)