mlsanityData to 7 Oct 2026

Revisiting Parameter-Efficient Tuning: Are We Really There Yet?

arXiv 2202.07962 ↗First posted 16 Feb 2022cs.CLcited by 76

Not on the map. Why?

Abstract (arXiv)

Parameter-Efficient Tuning (PETuning) methods have been deemed by many as the new paradigm for using pretrained language models (PLMs). By tuning just a fraction amount of parameters comparing to full model finetuning, PETuning methods claim to have achieved performance on par with or even better than finetuning. In this work, we take a step back and re-examine these PETuning methods by conducting the first comprehensive investigation into the training and evaluation of them. We found the problematic validation and testing practice in current studies, when accompanied by the instability nature of PETuning methods, has led to unreliable conclusions. When being compared under a truly fair evaluation protocol, PETuning cannot yield consistently competitive performance while finetuning remains to be the best-performing method in medium- and high-resource settings. We delve deeper into the cause of the instability and observed that the number of trainable parameters and training iterations are two main factors: reducing trainable parameters and prolonging training iterations may lead to higher stability in PETuning methods.

Key facts. Cited by 76 papers in the map (data through 7 Oct 2026). Builds on 40 of its 53 references that are in the map. First posted on arXiv 16 Feb 2022.

Builds on

40 of 53 references.

References · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  10. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  11. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  12. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
28 more
  1. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  2. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  3. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  4. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  5. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  6. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  7. 2021GPT Understands, Toocited by 715
  8. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  9. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  10. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  11. 2020AdapterHub: A Framework for Adapting Transformerscited by 441
  12. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 377
  13. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
  14. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 370
  15. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  16. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  17. 2020On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselinescited by 254
  18. 2020Revisiting Few-sample BERT Fine-tuningcited by 237
  19. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  20. 2021PPT: Pre-trained Prompt Tuning for Few-shot Learningcited by 232
  21. 2021UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuningcited by 162
  22. 2018Model Evaluation, Model Selection, and Algorithm Selection in Machine Learningcited by 157
  23. 2021On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptationcited by 147
  24. 2019Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Modelscited by 141
  25. 2017A Tutorial on Kernel Density Estimation and Recent Advancescited by 100
  26. 2019What Would Elsa Do? Freezing Layers During Transformer Fine-Tuningcited by 87
  27. 2021Robust Transfer Learning with Pretrained Language Models through Adapterscited by 35
  28. 2021Mixture-of-Partitions: Infusing Large Biomedical Knowledge Graphs into BERTcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 9; 2023: 23; 2024: 28; 2025: 10; 2026: 6.

76 papers.

Sort:

By area:

  1. 2023A Comprehensive Overview of Large Language Modelscited by 639
  2. 2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 140
  3. 2022Neural Prompt Searchcited by 133
  4. 2023Deep Model Fusion: A Surveycited by 89
  5. 2022When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methodscited by 74
  6. 2023Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytescited by 55
  7. 2022ColD Fusion: Collaborative Descent for Distributed Multitask Finetuningcited by 52
  8. 2024LLaVA Steering: Visual Instruction Tuning with 500x Fewer Parameters through Modality Linear Representation-Steeringcited by 50
  9. 2023AutoPEFT: Automatic Configuration Search for Parameter-Efficient Fine-Tuningcited by 49
  10. 2023Empirical Analysis of the Strengths and Weaknesses of PEFT Techniques for LLMscited by 38
  11. 2024BitDelta: Your Fine-Tune May Only Be Worth One Bitcited by 38
  12. 2024Personalized LLM Response Generation with Parameterized Memory Injectioncited by 33
38 more
  1. 2023SiRA: Sparse Mixture of Low Rank Adaptationcited by 32
  2. 2024Facing the Elephant in the Room: Visual Prompt Tuning or Full Finetuning?cited by 31
  3. 2024LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimizationcited by 31
  4. 2023Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applicationscited by 27
  5. 2023Neural Architecture Search for Parameter-Efficient Fine-tuning of Large Pre-trained Language Modelscited by 25
  6. 2022Where to start? Analyzing the potential value of intermediate modelscited by 23
  7. 2024Astraios: Parameter-Efficient Instruction Tuning Code Large Language Modelscited by 23
  8. 2024An Empirical Study on Parameter-Efficient Fine-Tuning for MultiModal Large Language Modelscited by 22
  9. 2024Understanding Linear Probing then Fine-tuning Language Models from NTK Perspectivecited by 21
  10. 2025Prompt-based Adaptation in Large-scale Vision Models: A Surveycited by 21
  11. 2024Erasing the Bias: Fine-Tuning Foundation Models for Semi-Supervised Learningcited by 19
  12. 2024Understanding Layer Significance in LLM Alignmentcited by 19
  13. 2024Test-Time Low Rank Adaptation via Confidence Maximization for Zero-Shot Generalization of Vision-Language Modelscited by 17
  14. 2022Sparse Structure Search for Parameter-Efficient Tuningcited by 16
  15. 2024Lessons and Insights from a Unifying Study of Parameter-Efficient Fine-Tuning (PEFT) in Visual Recognitioncited by 16
  16. 2022Parameter-Efficient Tuning Makes a Good Classification Headcited by 15
  17. 2023A Survey on Stability of Learning with Limited Labelled Data and its Sensitivity to the Effects of Randomnesscited by 14
  18. 2023Low-Rank Adaptation for Multilingual Summarization: An Empirical Studycited by 13
  19. 2023Dynamic Prompting: A Unified Framework for Prompt Tuningcited by 12
  20. 2023Efficient Storage of Fine-Tuned Models via Low-Rank Approximation of Weight Residualscited by 12
  21. 2024Quantized Delta Weight Is Safety Keepercited by 12
  22. 2025LoRASculpt: Sculpting LoRA for Harmonizing General and Specialized Knowledge in Multimodal Large Language Modelscited by 11
  23. 2025UORA: Uniform Orthogonal Reinitialization Adaptation in Parameter-Efficient Fine-Tuning of Large Modelscited by 11
  24. 2024RST-LoRA: A Discourse-Aware Low-Rank Adaptation for Long Document Abstractive Summarizationcited by 10
  25. 2024DARE the Extreme: Revisiting Delta-Parameter Pruning For Fine-Tuned Modelscited by 10
  26. 2023A Comprehensive Analysis of Adapter Efficiencycited by 8
  27. 2023Stitched ViTs are Flexible Vision Backbonescited by 8
  28. 2025Visual Variational Autoencoder Prompt Tuningcited by 8
  29. 2025All You Need is One: Capsule Prompt Tuning with a Single Vectorcited by 8
  30. 2022When does Parameter-Efficient Transfer Learning Work for Machine Translation?cited by 7
  31. 2024XFT: Unlocking the Power of Code Instruction Tuning by Simply Merging Upcycled Mixture-of-Expertscited by 7
  32. 2024Efficient and Versatile Robust Fine-Tuning of Zero-shot Modelscited by 7
  33. 2022Parameter-Efficient Finetuning of Transformers for Source Codecited by 6
  34. 2023A Language Model of Java Methods with Train/Test Deduplicationcited by 6
  35. 2023MixPHM: Redundancy-Aware Parameter-Efficient Tuning for Low-Resource Visual Question Answeringcited by 5
  36. 2023A Comprehensive Evaluation of Parameter-Efficient Fine-Tuning on Software Engineering Taskscited by 5
  37. 2024A Federated Learning-Friendly Approach for Parameter-Efficient Fine-Tuning of SAM in 3D Segmentationcited by 5
  38. 2024CleaR: Towards Robust and Generalized Parameter-Efficient Fine-Tuning for Noisy Label Learningcited by 5

50 most cited of 76 All 76 (JSON)