mlsanityData to 7 Oct 2026

Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment

arXiv 2312.12148 ↗First posted 19 Dec 2023cs.CLcited by 227

Not on the map. Why?

Abstract (arXiv)

With the continuous growth in the number of parameters of transformer-based pretrained language models (PLMs), particularly the emergence of large language models (LLMs) with billions of parameters, many natural language processing (NLP) tasks have demonstrated remarkable success. However, the enormous size and computational demands of these models pose significant challenges for adapting them to specific downstream tasks, especially in environments with limited computational resources. Parameter Efficient Fine-Tuning (PEFT) offers an effective solution by reducing the number of fine-tuning parameters and memory usage while achieving comparable performance to full fine-tuning. The demands for fine-tuning PLMs, especially LLMs, have led to a surge in the development of PEFT methods, as depicted in Fig. 1. In this paper, we present a comprehensive and systematic review of PEFT methods for PLMs. We summarize these PEFT methods, discuss their applications, and outline future directions. Furthermore, we conduct experiments using several representative PEFT methods to better understand their effectiveness in parameter efficiency and memory efficiency. By offering insights into the latest advancements and practical applications, this survey serves as an invaluable resource for researchers and practitioners seeking to navigate the challenges and opportunities presented by PEFT in the context of PLMs.

Key facts. Cited by 227 papers in the map (data through 7 Oct 2026). Builds on 98 of its 141 references that are in the map. First posted on arXiv 19 Dec 2023.

Builds on

98 of 141 references.

References · 98

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  9. 2016Layer Normalizationcited by 5,579
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  11. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  12. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
38 more
  1. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  3. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  4. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  5. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  6. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  7. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  8. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  9. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  10. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  11. 2022Editing Models with Task Arithmeticcited by 1,076
  12. 2016HyperNetworkscited by 1,003
  13. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  14. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  15. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  16. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  17. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  18. 2021GPT Understands, Toocited by 715
  19. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  20. 2017Learning multiple visual domains with residual adapterscited by 553
  21. 2019Importance Estimation for Neural Network Pruningcited by 531
  22. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  23. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
  24. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
  25. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
  26. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  27. 2017Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Sciencecited by 325
  28. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  29. 2019Revealing the Dark Secrets of BERTcited by 300
  30. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 278
  31. 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 239
  32. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  33. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 229
  34. 2022LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learningcited by 228
  35. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 223
  36. 2021WARP: Word-level Adversarial ReProgrammingcited by 220
  37. 2019Model Fusion via Optimal Transportcited by 217
  38. 2021Training Neural Networks with Fixed Sparse Maskscited by 212

50 most cited of 98 All 98 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 73; 2025: 86; 2026: 68.

227 papers.

Sort:

By area:

  1. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  2. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 622
  3. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 261
  4. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  5. 2024Let Your Graph Do the Talking: Encoding Structured Data for LLMscited by 112
  6. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  7. 2024Dual-Personalizing Adapter for Federated Foundation Modelscited by 54
  8. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 44
  9. 2024Combining Fine-Tuning and LLM-based Agents for Intuitive Smart Contract Auditing with Justificationscited by 41
  10. 2024CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuningcited by 41
  11. 2024LoRA Dropout as a Sparsity Regularizer for Overfitting Controlcited by 35
  12. 2025A Survey on Efficient Vision-Language-Action Modelscited by 34
38 more
  1. 2024How to build the best medical image segmentation algorithm using foundation models: a comprehensive empirical study with Segment Anything Modelcited by 32
  2. 2024A Review of Multi-Modal Large Language and Vision Modelscited by 30
  3. 2024Facial Affective Behavior Analysis with Instruction Tuningcited by 26
  4. 2024DiffCut: Catalyzing Zero-Shot Semantic Segmentation with Diffusion Features and Recursive Normalized Cutcited by 26
  5. 2024Compressible Dynamics in Deep Overparameterized Low-Rank Learning & Adaptationcited by 26
  6. 2024Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problemcited by 23
  7. 2024Cross-Modal Safety Alignment: Is textual unlearning all you need?cited by 22
  8. 2024Parameter-Efficient Fine-Tuning for Pre-Trained Vision Models: A Survey and Benchmarkcited by 21
  9. 2024Spectral Adapter: Fine-Tuning in Spectral Spacecited by 21
  10. 2025Transformer Copilot: Learning from The Mistake Log in LLM Fine-tuningcited by 21
  11. 2024In-Context Data Distillation with TabPFNcited by 19
  12. 2026VideoGPA: Distilling Geometry Priors for 3D-Consistent Video Generationcited by 19
  13. 2024PYRA: Parallel Yielding Re-Activation for Training-Inference Efficient Task Adaptationcited by 17
  14. 2024CLIP with Generative Latent Replay: a Strong Baseline for Incremental Learningcited by 17
  15. 2024Towards Generalizable Scene Change Detectioncited by 17
  16. 2025Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamicscited by 16
  17. 2024Bridging The Gap between Low-rank and Orthogonal Adaptation via Householder Reflection Adaptationcited by 15
  18. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 14
  19. 2024Task Prompt Vectors: Effective Initialization through Multi-Task Soft-Prompt Transfercited by 13
  20. 2025Gradient-Based Model Fingerprinting for LLM Similarity Detection and Family Classificationcited by 13
  21. 2024Adversarial Representation Engineering: A General Model Editing Framework for Large Language Modelscited by 12
  22. 2024Randomized Asymmetric Chain of LoRA: The First Meaningful Theoretical Framework for Low-Rank Adaptationcited by 12
  23. 2025LoX: Low-Rank Extrapolation Robustifies LLM Safety Against Fine-tuningcited by 12
  24. 2025Kvasir-VQA-x1: A Multimodal Dataset for Medical Reasoning and Robust MedVQA in Gastrointestinal Endoscopycited by 11
  25. 2024Automated Text Scoring in the Age of Generative AI for the GPU-poorcited by 10
  26. 2024CURLoRA: Stable LLM Continual Fine-Tuning and Catastrophic Forgetting Mitigationcited by 10
  27. 2024LLM4AD: Large Language Models for Autonomous Driving -- Concept, Review, Benchmark, Experiments, and Future Trendscited by 10
  28. 2024S2FT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsitycited by 10
  29. 2025LoRA-X: Bridging Foundation Models with Training-Free Cross-Model Adaptationcited by 10
  30. 2025Focus Directions Make Your Language Models Pay More Attention to Relevant Contextscited by 9
  31. 2025SOFT: Selective Data Obfuscation for Protecting LLM Fine-tuning against Membership Inference Attackscited by 9
  32. 2025QR-LoRA: Efficient and Disentangled Fine-tuning via QR Decomposition for Customized Generationcited by 9
  33. 2025A Guardrail for Safety Preservation: When Safety-Sensitive Subspace Meets Harmful-Resistant Null-Spacecited by 9
  34. 2026Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuningcited by 9
  35. 2024From Text to Life: On the Reciprocal Relationship between Artificial Life and Large Language Modelscited by 8
  36. 2024LLMs4OL 2024 Overview: The 1st Large Language Models for Ontology Learning Challengecited by 8
  37. 2024Simultaneous Computation and Memory Efficient Zeroth-Order Optimizer for Fine-Tuning Large Language Modelscited by 8
  38. 2024RoCoFT: Efficient Finetuning of Large Language Models with Row-Column Updatescited by 8

50 most cited of 227 All 227 (JSON)