mlsanityData to 7 Oct 2026

Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey

arXiv 2403.14608 ↗First posted 21 Mar 2024cs.LGcited by 641

Not on the map. Why?

Cited by 34 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large models represent a groundbreaking advancement in multiple application fields, enabling remarkable achievements across various tasks. However, their unprecedented scale comes with significant computational costs. These models, often consisting of billions of parameters, require vast amounts of computational resources for execution. Especially, the expansive scale and computational demands pose considerable challenges when customizing them for particular downstream tasks, particularly over the hardware platforms constrained by computational capabilities. Parameter Efficient Fine-Tuning (PEFT) provides a practical solution by efficiently adjusting the large models over the various downstream tasks. In particular, PEFT refers to the process of adjusting the parameters of a pre-trained large model to adapt it to a specific task or domain while minimizing the number of additional parameters introduced or computational resources required. This approach is particularly important when dealing with large-scale language models with high parameter counts, as fine-tuning these models from scratch can be computationally expensive and resource-intensive, posing considerable challenges in the supporting system platform design. In this survey, we present comprehensive studies of various PEFT algorithms, examining their performance and computational overhead. Moreover, we provide an overview of applications developed using different PEFT algorithms and discuss common techniques employed to mitigate computation costs for PEFT. In addition to providing an extensive survey from an algorithmic standpoint, we also examine various real-world system designs to investigate the implementation costs associated with different PEFT approaches. This survey serves as a valuable resource for researchers aiming to understand both the PEFT algorithm and its system implementation, offering detailed ......

Key facts. Cited by 641 papers in the map (data through 7 Oct 2026). Builds on 209 of its 264 references that are in the map. First posted on arXiv 21 Mar 2024.

Builds on

209 of 264 references.

References · 209

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  11. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  12. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
38 more
  1. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  10. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  12. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  13. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  14. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  15. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  16. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  17. 2015VQA: Visual Question Answeringcited by 2,489
  18. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  19. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  20. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  21. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  22. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  23. 2017The Kinetics Human Action Video Datasetcited by 1,894
  24. 2022Visual Prompt Tuningcited by 1,674
  25. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  26. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
  27. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  28. 2019Root Mean Square Layer Normalizationcited by 1,425
  29. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,379
  30. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  31. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  32. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  33. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  34. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
  35. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  36. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  37. 2022Diffusion Models in Vision: A Surveycited by 1,008
  38. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 958

50 most cited of 209 All 209 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 156; 2025: 297; 2026: 188.

641 papers.

Sort:

By area:

  1. 2025O1-Pruner: Length-Harmonizing Fine-Tuning for O1-Like Reasoning Pruningcited by 278
  2. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 261
  3. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  4. 2024Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?cited by 227
  5. 2024ReFT: Representation Finetuning for Language Modelscited by 186
  6. 2024ControlNeXt: Powerful and Efficient Control for Image and Video Generationcited by 158
  7. 2024LoRA-GA: Low-Rank Adaptation with Gradient Approximationcited by 134
  8. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  9. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  10. 2024A Review of Large Language Models and Autonomous Agents in Chemistrycited by 96
  11. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 83
  12. 2024LLMs Will Always Hallucinate, and We Need to Live With Thiscited by 77
38 more
  1. 2024A Survey on LoRA of Large Language Modelscited by 67
  2. 2024A Survey on Employing Large Language Models for Text-to-SQL Taskscited by 67
  3. 2024Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wiselycited by 65
  4. 2025Large language models for automated scholarly paper review: A surveycited by 59
  5. 2024The Ninth NTIRE 2024 Efficient Super-Resolution Challenge Reportcited by 57
  6. 2024Mixture-of-Subspaces in Low-Rank Adaptationcited by 47
  7. 2024Recent Advances of Foundation Language Models-based Continual Learning: A Surveycited by 46
  8. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 44
  9. 20243D-HGS: 3D Half-Gaussian Splattingcited by 43
  10. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 43
  11. 2024LoRA-FAIR: Federated LoRA Fine-Tuning with Aggregation and Initialization Refinementcited by 37
  12. 2024LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statisticscited by 35
  13. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 35
  14. 2024A Survey on Medical Large Language Models: Technology, Application, Trustworthiness, and Future Directionscited by 34
  15. 2025RandLoRA: Full-rank parameter-efficient fine-tuning of large modelscited by 34
  16. 2025TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problemscited by 33
  17. 2024A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithmscited by 30
  18. 2024ModalPrompt: Towards Efficient Multimodal Continual Instruction Tuning with Dual-Modality Guided Promptcited by 29
  19. 2025Parallel Scaling Law for Language Modelscited by 29
  20. 2024Large Model Based Agents: State-of-the-Art, Cooperation Paradigms, Security and Privacy, and Future Trendscited by 26
  21. 2024Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Passcited by 26
  22. 2025Can Masked Autoencoders Also Listen to Birds?cited by 26
  23. 2024Parameter Efficient Fine Tuning: A Comprehensive Analysis Across Applicationscited by 25
  24. 2024CrisisSense-LLM: Instruction Fine-Tuned Large Language Model for Multi-label Social Media Text Classification in Disaster Informaticscited by 25
  25. 2024A Survey on Large Language Models for Communication, Network, and Service Management: Application Insights, Challenges, and Future Directionscited by 25
  26. 2025RobustMerge: Parameter-Efficient Model Merging for MLLMs with Direction Robustnesscited by 25
  27. 2025The Mirage of Model Editing: Revisiting Evaluation in the Wildcited by 24
  28. 2025Tina: Tiny Reasoning Models via LoRAcited by 23
  29. 2024Remote Sensing SpatioTemporal Vision-Language Models: A Comprehensive Surveycited by 22
  30. 2024From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalitiescited by 22
  31. 2025AuroRA: Breaking Low-Rank Bottleneck of LoRA with Nonlinear Mappingcited by 22
  32. 2024Networking Systems for Video Anomaly Detection: A Tutorial and Surveycited by 21
  33. 2024Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Modelscited by 21
  34. 2024Learning on LoRAs: GL-Equivariant Processing of Low-Rank Weight Spaces for Large Finetuned Modelscited by 21
  35. 2024Mitigating the Risk of Health Inequity Exacerbated by Large Language Modelscited by 21
  36. 2025GSQ-Tuning: Group-Shared Exponents Integer in Fully Quantized Training for LLMs On-Device Fine-tuningcited by 21
  37. 2025Transformer Copilot: Learning from The Mistake Log in LLM Fine-tuningcited by 21
  38. 2024RoseLoRA: Row and Column-wise Sparse Low-rank Adaptation of Pre-trained Language Model for Knowledge Editing and Fine-tuningcited by 20

50 most cited of 641 All 641 (JSON)