# Parameter-Efficient Fine-Tuning for Foundation Models

arXiv: https://arxiv.org/abs/2501.13787 · HTML: https://mlsanity.com/paper/2501.13787/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Builds on 235 of its 308 references that are in the map.
- First posted on arXiv 23 Jan 2025.

## Builds on (most cited first, 30 of 235)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)

## Built on by (most cited first, 30 of 45)

- [A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Models](https://mlsanity.com/paper/2502.15828/) (2025)
- [TDRM: Smooth Reward Models with Temporal Difference for LLM RL and Inference](https://mlsanity.com/paper/2509.15110/) (2025)
- [The Primacy of Magnitude in Low-Rank Adaptation](https://mlsanity.com/paper/2507.06558/) (2025)
- [FG-OrIU: Towards Better Forgetting via Feature-Gradient Orthogonality for Incremental Unlearning](https://mlsanity.com/paper/2601.13578/) (2026)
- [CURE:Circuit-Aware Unlearning for LLM-based Recommendation](https://mlsanity.com/paper/2604.04982/) (2026)
- [Prompt to Polyp: Medical Text-Conditioned Image Synthesis with Diffusion Models](https://mlsanity.com/paper/2505.05573/) (2025)
- [Evolution of meta's llama models and parameter-efficient fine-tuning of large language models: a survey](https://mlsanity.com/paper/2510.12178/) (2025)
- [Benchmarking Foundation Models and Parameter-Efficient Fine-Tuning for Prognosis Prediction in Medical Imaging](https://mlsanity.com/paper/2506.18434/) (2025)
- [PASs-MoE: Mitigating Misaligned Co-drift among Router and Experts via Pathway Activation Subspaces for Continual Learning](https://mlsanity.com/paper/2601.13020/) (2026)
- [Segment Any Crack: Deep Semantic Segmentation Adaptation for Crack Detection](https://mlsanity.com/paper/2504.14138/) (2025)
- [Branch, or Layer? Zeroth-Order Optimization for Continual Learning of Vision-Language Models](https://mlsanity.com/paper/2506.12409/) (2025)
- [Task-Adaptive Parameter-Efficient Fine-Tuning for Weather Foundation Models](https://mlsanity.com/paper/2509.22020/) (2025)
- [Tadpole: Autoencoders as Foundation Models for 3D PDEs with Online Learning](https://mlsanity.com/paper/2605.15284/) (2026)
- [Token Adaptation via Side Graph Convolution for Efficient Fine-tuning of 3D Point Cloud Transformers](https://mlsanity.com/paper/2502.14142/) (2025)
- [Taming LLMs by Scaling Learning Rates with Gradient Grouping](https://mlsanity.com/paper/2506.01049/) (2025)
- [Perspective on Utilizing Foundation Models for Laboratory Automation in Materials Research](https://mlsanity.com/paper/2506.12312/) (2025)
- [TAP: Two-Stage Adaptive Personalization of Multi-Task and Multi-Modal Foundation Models in Federated Learning](https://mlsanity.com/paper/2509.26524/) (2025)
- [Parameter Importance-Driven Continual Learning for Foundation Models](https://mlsanity.com/paper/2511.15375/) (2025)
- [EvoEdit: Lifelong Free-Text Knowledge Editing through Latent Perturbation Augmentation and Knowledge-driven Parameter Fusion](https://mlsanity.com/paper/2512.04545/) (2025)
- [Surgical AI Copilot: Energy-Based Fourier Gradient Low-Rank Adaptation for Surgical LLM Agent Reasoning and Planning](https://mlsanity.com/paper/2503.09474/) (2025)
- [C-Flat++: Towards a More Efficient and Powerful Framework for Continual Learning](https://mlsanity.com/paper/2508.18860/) (2025)
- [MediRec: Enhancing Chinese Medication Recommendation with Explainable Clinical Reasoning](https://mlsanity.com/paper/2510.21084/) (2025)
- [Unleashing the Power of Vision-Language Models for Long-Tailed Multi-Label Visual Recognition](https://mlsanity.com/paper/2511.20641/) (2025)
- [RotMoLE: Enhancing Mixture of Low-Rank Experts through Rotational Gating Mechanism](https://mlsanity.com/paper/2605.25565/) (2026)
- [S2A: A Unified Framework for Parameter and Memory Efficient Transfer Learning](https://mlsanity.com/paper/2503.08154/) (2025)
- [Lung Nodule-SSM: Self-Supervised Lung Nodule Detection and Classification in Thoracic CT Images](https://mlsanity.com/paper/2505.15120/) (2025)
- [IPA: An Information-Reconstructive Input Projection Framework for Efficient Foundation Model Adaptation](https://mlsanity.com/paper/2509.04398/) (2025)
- [HiCoLoRA: Addressing Context-Prompt Misalignment via Hierarchical Collaborative LoRA for Zero-Shot DST](https://mlsanity.com/paper/2509.19742/) (2025)
- [CrossEarth-Gate: Fisher-Guided Adaptive Tuning Engine for Efficient Adaptation of Cross-Domain Remote Sensing Semantic Segmentation](https://mlsanity.com/paper/2511.20302/) (2025)
- [Rethinking Fine-Tuning: Unlocking Hidden Capabilities in Vision-Language Models](https://mlsanity.com/paper/2512.23073/) (2025)

## Abstract (arXiv)

This survey delves into the realm of Parameter-Efficient Fine-Tuning (PEFT) within the context of Foundation Models (FMs). PEFT, a cost-effective fine-tuning technique, minimizes parameters and computational complexity while striving for optimal downstream task performance. FMs, like ChatGPT, DALL-E, and LLaVA specialize in language understanding, generative tasks, and multimodal tasks, trained on diverse datasets spanning text, images, and videos. The diversity of FMs guides various adaptation strategies for PEFT. Therefore, this survey aims to provide a comprehensive overview of PEFT techniques applied to diverse FMs and address critical gaps in understanding the techniques, trends, and applications. We start by providing a detailed development of FMs and PEFT. Subsequently, we systematically review the key categories and core mechanisms of PEFT across diverse FMs to offer a comprehensive understanding of trends. We also explore the most recent applications across various FMs to demonstrate the versatility of PEFT, shedding light on the integration of systematic PEFT methods with a range of FMs. Furthermore, we identify potential research and development directions for improving PEFTs in the future. This survey provides a valuable resource for both newcomers and experts seeking to understand and use the power of PEFT across FMs. All reviewed papers are listed at urlhttps://github.com/THUDM/Awesome-Parameter-Efficient-Fine-Tuning-for-Foundation-Models.