mlsanityData to 7 Oct 2026

Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning

arXiv 2303.15647 ↗First posted 28 Mar 2023cs.CLcited by 171

Not on the map. Why?

Abstract (arXiv)

This paper presents a systematic overview of parameter-efficient fine-tuning methods, covering over 50 papers published between early 2019 and mid-2024. These methods aim to address the challenges of fine-tuning large language models by training only a small subset of parameters. We provide a taxonomy that covers a broad range of methods and present a detailed method comparison with a specific focus on real-life efficiency in fine-tuning multibillion-scale language models. We also conduct an extensive head-to-head experimental comparison of 15 diverse PEFT methods, evaluating their performance and efficiency on models up to 11B parameters. Our findings reveal that methods previously shown to surpass a strong LoRA baseline face difficulties in resource-constrained settings, where hyperparameter optimization is limited and the network is fine-tuned only for a few epochs. Finally, we provide a set of practical recommendations for using PEFT methods and outline potential future research directions.

Key facts. Cited by 171 papers in the map (data through 7 Oct 2026). Builds on 59 of its 76 references that are in the map. First posted on arXiv 28 Mar 2023.

Builds on

59 of 76 references.

References · 59

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  9. 2016Layer Normalizationcited by 5,579
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  11. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  12. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
38 more
  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  4. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  6. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  7. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  8. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  9. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  10. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  11. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  12. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  13. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  14. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  15. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  16. 2021GPT Understands, Toocited by 715
  17. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  18. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  19. 2017Learning multiple visual domains with residual adapterscited by 553
  20. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  21. 2020AdapterHub: A Framework for Adapting Transformerscited by 441
  22. 2013DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognitioncited by 422
  23. 2015Deep Learning with Limited Numerical Precisioncited by 368
  24. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  25. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  26. 2018Stronger generalization bounds for deep nets via a compression approachcited by 306
  27. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 247
  28. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  29. 2022LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learningcited by 228
  30. 2021WARP: Word-level Adversarial ReProgrammingcited by 220
  31. 2021Training Neural Networks with Fixed Sparse Maskscited by 212
  32. 2018Efficient parametrization of multi-domain deep neural networkscited by 168
  33. 2021UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuningcited by 162
  34. 2021Composable Sparse Fine-Tuning for Cross-Lingual Transfercited by 144
  35. 2022KronA: Parameter Efficient Tuning with Kronecker Adaptercited by 130
  36. 2022A Kernel-Based View of Language Model Fine-Tuningcited by 118
  37. 2022SparseAdapter: An Easy Approach for Improving the Parameter-Efficiency of Adapterscited by 83
  38. 2023Modular Deep Learningcited by 82

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 34; 2024: 63; 2025: 45; 2026: 29.

171 papers.

Sort:

By area:

  1. 2023Adapting Language Models to Compress Contextscited by 350
  2. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  3. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 227
  4. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  5. 2023Parameter-Efficient Orthogonal Finetuning via Butterfly Factorizationcited by 112
  6. 2023Mechanistically analyzing the effects of fine-tuning on procedurally defined taskscited by 110
  7. 2024Low-Rank Few-Shot Adaptation of Vision-Language Modelscited by 103
  8. 2024MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuningcited by 89
  9. 2023MA-SAM: Modality-agnostic SAM Adaptation for 3D Medical Image Segmentationcited by 82
  10. 2024Asymmetry in Low-Rank Adapters of Foundation Modelscited by 81
  11. 2024A Survey of Large Language Models in Finance (FinLLMs)cited by 77
  12. 2024Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directionscited by 77
38 more
  1. 2023Efficient Benchmarking of Language Modelscited by 71
  2. 2024Tracking Meets LoRA: Faster Training, Larger Model, Stronger Performancecited by 70
  3. 2024Learning to Route Among Specialized Experts for Zero-Shot Generalizationcited by 65
  4. 2023Adapters: A Unified Library for Parameter-Efficient and Modular Transfer Learningcited by 53
  5. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 44
  6. 2023When Do Prompting and Prefix-Tuning Work? A Theory of Capabilities and Limitationscited by 41
  7. 2024Recent Advances of Multimodal Continual Learning: A Comprehensive Surveycited by 39
  8. 2024LoRA Dropout as a Sparsity Regularizer for Overfitting Controlcited by 35
  9. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 35
  10. 2023DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuningcited by 34
  11. 2025TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problemscited by 33
  12. 2024APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inferencecited by 29
  13. 2024Time-, Memory- and Parameter-Efficient Visual Adaptationcited by 28
  14. 2023Ladder Fine-tuning approach for SAM integrating complementary networkcited by 25
  15. 2024Scaling Sparse Fine-Tuning to Large Language Modelscited by 25
  16. 2024Prompting a Pretrained Transformer Can Be a Universal Approximatorcited by 25
  17. 2024Dynamic Tuning Towards Parameter and Inference Efficiency for ViT Adaptationcited by 25
  18. 2024Compress then Serve: Serving Thousands of LoRA Adapters with Little Overheadcited by 24
  19. 2024T-RAG: Lessons from the LLM Trenchescited by 23
  20. 2023Model Compression in Practice: Lessons Learned from Practitioners Creating On-device Machine Learning Experiencescited by 21
  21. 2024Parameter Efficient Quasi-Orthogonal Fine-Tuning via Givens Rotationcited by 21
  22. 2025Rethinking Few-Shot Adaptation of Vision-Language Models in Two Stagescited by 17
  23. 2023Influence of External Information on Large Language Models Mirrors Social Cognitive Patternscited by 16
  24. 2024Random Masking Finds Winning Tickets for Parameter Efficient Fine-tuningcited by 16
  25. 2024A Survey on Large Language Models from General Purpose to Medical Applications: Datasets, Methodologies, and Evaluationscited by 16
  26. 2024Lessons and Insights from a Unifying Study of Parameter-Efficient Fine-Tuning (PEFT) in Visual Recognitioncited by 16
  27. 2024Revisiting Prefix-tuning: Statistical Benefits of Reparameterization among Promptscited by 15
  28. 2023Applications of Large Scale Foundation Models for Autonomous Drivingcited by 14
  29. 2023PERP: Rethinking the Prune-Retrain Paradigm in the Era of LLMscited by 14
  30. 2024Synergizing Foundation Models and Federated Learning: A Surveycited by 14
  31. 2023Realistic Unsupervised CLIP Fine-tuning with Universal Entropy Optimizationcited by 13
  32. 2023Fine-Tuning Llama 2 Large Language Models for Detecting Online Sexual Predatory Chats and Abusive Textscited by 13
  33. 2023When Geoscience Meets Foundation Models: Towards General Geoscience Artificial Intelligence Systemcited by 13
  34. 2023Asking Before Acting: Gather Information in Embodied Decision Making with Language Modelscited by 12
  35. 2024HiFT: A Hierarchical Full Parameter Fine-Tuning Strategycited by 12
  36. 2024FreeSeg-Diff: Training-Free Open-Vocabulary Segmentation with Diffusion Modelscited by 12
  37. 2024Human-inspired Perspectives: A Survey on AI Long-term Memorycited by 12
  38. 2024MambaPEFT: Exploring Parameter-Efficient Fine-Tuning for Mambacited by 11

50 most cited of 171 All 171 (JSON)