mlsanityData to 7 Oct 2026

UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuning

arXiv 2110.07577 ↗First posted 14 Oct 2021cs.CL cs.AI cs.LGcited by 162

Not on the map. Why?

Abstract (arXiv)

Recent parameter-efficient language model tuning (PELT) methods manage to match the performance of fine-tuning with much fewer trainable parameters and perform especially well when training data is limited. However, different PELT methods may perform rather differently on the same task, making it nontrivial to select the most appropriate method for a specific task, especially considering the fast-growing number of new PELT methods and tasks. In light of model diversity and the difficulty of model selection, we propose a unified framework, UniPELT, which incorporates different PELT methods as submodules and learns to activate the ones that best suit the current data or task setup via gating mechanism. On the GLUE benchmark, UniPELT consistently achieves 1~4% gains compared to the best individual PELT method that it incorporates and even outperforms fine-tuning under different setups. Moreover, UniPELT generally surpasses the upper bound that takes the best performance of all its submodules used individually on each task, indicating that a mixture of multiple PELT methods may be inherently more effective than single methods.

Key facts. Cited by 162 papers in the map (data through 7 Oct 2026). Builds on 23 of its 27 references that are in the map. First posted on arXiv 14 Oct 2021.

Builds on

23 of 27 references.

References · 23

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  9. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  10. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  11. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  12. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
11 more
  1. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  2. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  3. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  4. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
  5. 2020AdapterHub: A Framework for Adapting Transformerscited by 441
  6. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  7. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 278
  8. 2021PPT: Pre-trained Prompt Tuning for Few-shot Learningcited by 232
  9. 2021On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptationcited by 147
  10. 2021DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learningcited by 114
  11. 2019What Would Elsa Do? Freezing Layers During Transformer Fine-Tuningcited by 87

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 19; 2023: 54; 2024: 48; 2025: 23; 2026: 15.

162 papers.

Sort:

By area:

  1. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  2. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  3. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  4. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 317
  5. 2022Reasoning with Language Model Prompting: A Surveycited by 251
  6. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 229
  7. 2022LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learningcited by 228
  8. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 227
  9. 2023Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuningcited by 171
  10. 2023RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensingcited by 150
  11. 2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 140
  12. 2023Continual Diffusion: Continual Customization of Text-to-Image Diffusion with C-LoRAcited by 138
38 more
  1. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
  2. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
  3. 2022Neural Prompt Searchcited by 133
  4. 2023Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learningcited by 115
  5. 2023Parameter-Efficient Orthogonal Finetuning via Butterfly Factorizationcited by 112
  6. 2023Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantizationcited by 97
  7. 2022BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Promptingcited by 90
  8. 2022Revisiting Parameter-Efficient Tuning: Are We Really There Yet?cited by 76
  9. 2022StoryDALL-E: Adapting Pretrained Text-to-Image Transformers for Story Continuationcited by 74
  10. 2023Parameter-Efficient Fine-Tuning Design Spacescited by 67
  11. 2022Visual Query Tuning: Towards Effective Usage of Intermediate Representations for Parameter and Memory Efficient Transfer Learningcited by 57
  12. 2023Adapters: A Unified Library for Parameter-Efficient and Modular Transfer Learningcited by 53
  13. 2024Dynamic Adapter Meets Prompt Tuning: Parameter-Efficient Transfer Learning for Point Cloud Analysiscited by 52
  14. 2023AutoPEFT: Automatic Configuration Search for Parameter-Efficient Fine-Tuningcited by 49
  15. 2023TAIL: Task-specific Adapters for Imitation Learning with Large Pretrained Modelscited by 48
  16. 2023IncreLoRA: Incremental Parameter Allocation Method for Parameter-Efficient Fine-tuningcited by 45
  17. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 44
  18. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 43
  19. 2023Empirical Analysis of the Strengths and Weaknesses of PEFT Techniques for LLMscited by 38
  20. 2023FAME-ViL: Multi-Tasking Vision-Language Model for Heterogeneous Fashion Taskscited by 37
  21. 2024Advancing Parameter Efficiency in Fine-tuning via Representation Editingcited by 37
  22. 2023Visual Tuningcited by 34
  23. 2023DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuningcited by 34
  24. 2025TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problemscited by 33
  25. 2023SiRA: Sparse Mixture of Low Rank Adaptationcited by 32
  26. 2023MultiLoRA: Democratizing LoRA for Better Multi-Task Learningcited by 32
  27. 2023Batched Low-Rank Adaptation of Foundation Modelscited by 32
  28. 2021FedHM: Efficient Federated Learning for Heterogeneous Models via Low-rank Factorizationcited by 31
  29. 2023Learning to Modulate pre-trained Models in RLcited by 28
  30. 2023ControlNet-XS: Rethinking the Control of Text-to-Image Diffusion Models as Feedback-Control Systemscited by 28
  31. 2024Time-, Memory- and Parameter-Efficient Visual Adaptationcited by 28
  32. 2023Make Pre-trained Model Reversible: From Parameter to Memory Efficient Fine-Tuningcited by 26
  33. 2024QDyLoRA: Quantized Dynamic Low-Rank Adaptation for Efficient Large Language Model Tuningcited by 26
  34. 2024Facial Affective Behavior Analysis with Instruction Tuningcited by 26
  35. 2023G-Adapter: Towards Structure-Aware Parameter-Efficient Transfer Learning for Graph Transformer Networkscited by 25
  36. 2023Neural Architecture Search for Parameter-Efficient Fine-tuning of Large Pre-trained Language Modelscited by 25
  37. 2023VLN-PETL: Parameter-Efficient Transfer Learning for Vision-and-Language Navigationcited by 22
  38. 2025AuroRA: Breaking Low-Rank Bottleneck of LoRA with Nonlinear Mappingcited by 22

50 most cited of 162 All 162 (JSON)