mlsanityData to 14 Oct 2026

Parameter-Efficient Fine-Tuning for Foundation Models

arXiv 2501.13787 ↗First posted 23 Jan 2025cs.CL cs.AI cs.LGcited by 45

Not on the map. Why?

Abstract (arXiv)

This survey delves into the realm of Parameter-Efficient Fine-Tuning (PEFT) within the context of Foundation Models (FMs). PEFT, a cost-effective fine-tuning technique, minimizes parameters and computational complexity while striving for optimal downstream task performance. FMs, like ChatGPT, DALL-E, and LLaVA specialize in language understanding, generative tasks, and multimodal tasks, trained on diverse datasets spanning text, images, and videos. The diversity of FMs guides various adaptation strategies for PEFT. Therefore, this survey aims to provide a comprehensive overview of PEFT techniques applied to diverse FMs and address critical gaps in understanding the techniques, trends, and applications. We start by providing a detailed development of FMs and PEFT. Subsequently, we systematically review the key categories and core mechanisms of PEFT across diverse FMs to offer a comprehensive understanding of trends. We also explore the most recent applications across various FMs to demonstrate the versatility of PEFT, shedding light on the integration of systematic PEFT methods with a range of FMs. Furthermore, we identify potential research and development directions for improving PEFTs in the future. This survey provides a valuable resource for both newcomers and experts seeking to understand and use the power of PEFT across FMs. All reviewed papers are listed at urlhttps://github.com/THUDM/Awesome-Parameter-Efficient-Fine-Tuning-for-Foundation-Models.

Key facts. Cited by 45 papers in the map (data through 14 Oct 2026). Builds on 235 of its 308 references that are in the map. First posted on arXiv 23 Jan 2025.

Builds on

235 of 308 references.

References · 235

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2023GPT-4 Technical Reportcited by 18,396
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  8. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  9. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  10. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  12. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
38 more
  1. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  3. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  9. 2023Qwen Technical Reportcited by 3,344
  10. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  11. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  12. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  13. 2023A Survey of Large Language Modelscited by 2,576
  14. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  15. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,261
  16. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  17. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  18. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  19. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  20. 2022Visual Prompt Tuningcited by 1,681
  21. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,489
  22. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,464
  23. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  24. 2019SuperGlue: Learning Feature Matching with Graph Neural Networkscited by 1,443
  25. 2015Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcastingcited by 1,436
  26. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,351
  27. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  28. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  29. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  30. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  31. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  32. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  33. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  34. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,026
  35. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
  36. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 961
  37. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 938
  38. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872

50 most cited of 235 All 235 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 25; 2026: 20.

45 papers.

Sort:

By area:

  1. 2025A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Modelscited by 9
  2. 2025TDRM: Smooth Reward Models with Temporal Difference for LLM RL and Inferencecited by 8
  3. 2025The Primacy of Magnitude in Low-Rank Adaptationcited by 7
  4. 2026FG-OrIU: Towards Better Forgetting via Feature-Gradient Orthogonality for Incremental Unlearningcited by 6
  5. 2026CURE:Circuit-Aware Unlearning for LLM-based Recommendationcited by 6
  6. 2025Prompt to Polyp: Medical Text-Conditioned Image Synthesis with Diffusion Modelscited by 5
  7. 2025Evolution of meta's llama models and parameter-efficient fine-tuning of large language models: a surveycited by 5
  8. 2025Benchmarking Foundation Models and Parameter-Efficient Fine-Tuning for Prognosis Prediction in Medical Imagingcited by 4
  9. 2026PASs-MoE: Mitigating Misaligned Co-drift among Router and Experts via Pathway Activation Subspaces for Continual Learningcited by 4
  10. 2025Segment Any Crack: Deep Semantic Segmentation Adaptation for Crack Detectioncited by 3
  11. 2025Branch, or Layer? Zeroth-Order Optimization for Continual Learning of Vision-Language Modelscited by 3
  12. 2025Task-Adaptive Parameter-Efficient Fine-Tuning for Weather Foundation Modelscited by 3
33 more
  1. 2026Tadpole: Autoencoders as Foundation Models for 3D PDEs with Online Learningcited by 3
  2. 2025Token Adaptation via Side Graph Convolution for Efficient Fine-tuning of 3D Point Cloud Transformerscited by 2
  3. 2025Taming LLMs by Scaling Learning Rates with Gradient Groupingcited by 2
  4. 2025Perspective on Utilizing Foundation Models for Laboratory Automation in Materials Researchcited by 2
  5. 2025TAP: Two-Stage Adaptive Personalization of Multi-Task and Multi-Modal Foundation Models in Federated Learningcited by 2
  6. 2025Parameter Importance-Driven Continual Learning for Foundation Modelscited by 2
  7. 2025EvoEdit: Lifelong Free-Text Knowledge Editing through Latent Perturbation Augmentation and Knowledge-driven Parameter Fusioncited by 2
  8. 2025Surgical AI Copilot: Energy-Based Fourier Gradient Low-Rank Adaptation for Surgical LLM Agent Reasoning and Planningcited by 1
  9. 2025C-Flat++: Towards a More Efficient and Powerful Framework for Continual Learningcited by 1
  10. 2025MediRec: Enhancing Chinese Medication Recommendation with Explainable Clinical Reasoningcited by 1
  11. 2025Unleashing the Power of Vision-Language Models for Long-Tailed Multi-Label Visual Recognitioncited by 1
  12. 2026RotMoLE: Enhancing Mixture of Low-Rank Experts through Rotational Gating Mechanismcited by 1
  13. 2025S2A: A Unified Framework for Parameter and Memory Efficient Transfer Learningcited by 0
  14. 2025Lung Nodule-SSM: Self-Supervised Lung Nodule Detection and Classification in Thoracic CT Imagescited by 0
  15. 2025IPA: An Information-Reconstructive Input Projection Framework for Efficient Foundation Model Adaptationcited by 0
  16. 2025HiCoLoRA: Addressing Context-Prompt Misalignment via Hierarchical Collaborative LoRA for Zero-Shot DSTcited by 0
  17. 2025CrossEarth-Gate: Fisher-Guided Adaptive Tuning Engine for Efficient Adaptation of Cross-Domain Remote Sensing Semantic Segmentationcited by 0
  18. 2025Rethinking Fine-Tuning: Unlocking Hidden Capabilities in Vision-Language Modelscited by 0
  19. 2026Enhancing Vision Language Models with Logic Reasoning for Situational Awarenesscited by 0
  20. 2026Shifting Adaptation from Weight Space to Memory Space: A Memory-Augmented Agent for Medical Image Segmentationcited by 0
  21. 2026IntentVLM: Open-Vocabulary Intention Recognition through Forward-Inverse Modeling with Video-Language Modelscited by 0
  22. 2026Fisher-Guided Progressive Parameter Selection for Adaptive Fine-Tuningcited by 0
  23. 2026The Hidden Power of Scaling Factor in LoRA Optimizationcited by 0
  24. 2026Data-Efficient Adaptation of LLMs via Attention Head Reweightingcited by 0
  25. 2026Energy- and Memory-Efficient PEFT Methods for Personalized On-Device SLMs on Consumer GPUscited by 0
  26. 2026APQF: Agentic Profiling-Guided Structured Pruning and Mixed-Precision Quantization with Adaptive Fine-Tuningcited by 0
  27. 2026Fine-Tuning of Transformer models with Framescited by 0
  28. 2026FLoKD: Adaptive Knowledge Distillation for Federated Low-Rank LLM over Wireless Networkscited by 0
  29. 20260.5%>100%: Bidirectional Reciprocal Learning for Referring Image Segmentationcited by 0
  30. 2026Towards Understanding LLM-Based Log Anomaly Detection: An Empirical Study of Performance, Efficiency, and Robustnesscited by 0
  31. 2026Fisher-Informed Recalibration for Feedback-Based On-Policy Self-Distillation of LLMscited by 0
  32. 2026Masked Feature Encoding for Large-Scale Whole Slide Image Representationcited by 0
  33. 2026Strategic Governance of AI Models in Earth Sciencecited by 0