# PEFT A2Z: Parameter-Efficient Fine-Tuning Survey for Large Language and Vision Models

arXiv: https://arxiv.org/abs/2504.14117 · HTML: https://mlsanity.com/paper/2504.14117/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- First posted on arXiv 19 Apr 2025.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Built on by (most cited first, 16 of 16)

- [Not All Directions Matter: Towards Structured and Task-Aware Low-Rank Model Adaptation](https://mlsanity.com/paper/2603.14228/) (2026)
- [Evolution of meta's llama models and parameter-efficient fine-tuning of large language models: a survey](https://mlsanity.com/paper/2510.12178/) (2025)
- [PEFT-Bench: A Parameter-Efficient Fine-Tuning Methods Benchmark](https://mlsanity.com/paper/2511.21285/) (2025)
- [TASO: Task-Aligned Sparse Optimization for Parameter-Efficient Model Adaptation](https://mlsanity.com/paper/2509.17688/) (2025)
- [Evolutionary Multi-Task Optimization for LLM-Guided Program Discovery](https://mlsanity.com/paper/2605.22613/) (2026)
- [LoKI: Low-damage Knowledge Implanting of Large Language Models](https://mlsanity.com/paper/2505.22120/) (2025)
- [SliceFine: The Universal Winning-Slice Hypothesis for Pretrained Networks](https://mlsanity.com/paper/2510.08513/) (2025)
- [Monkey Jump : MoE-Style PEFT for Efficient Multi-Task Learning](https://mlsanity.com/paper/2601.06356/) (2026)
- [Adaptive parameter-efficient fine-tuning via Hessian-informed subset selection](https://mlsanity.com/paper/2505.12579/) (2025)
- [VT-LVLM-AR: A Video-Temporal Large Vision-Language Model Adapter for Fine-Grained Action Recognition in Long-Term Videos](https://mlsanity.com/paper/2508.15903/) (2025)
- [An Empirical Investigation of Pre-Trained Deep Learning Model Reuse in the Scientific Process](https://mlsanity.com/paper/2603.13584/) (2026)
- [Frequency Switching Mechanism for Parameter-E!cient Multi-Task Learning](https://mlsanity.com/paper/2603.21111/) (2026)
- [Beyond the Linear Separability Ceiling: Aligning Representations in VLMs](https://mlsanity.com/paper/2507.07574/) (2025)
- [A Foundation Model for DAS Signal Recognition and Visual Prompt Tuning of the Pre-trained Model for Downstream Tasks](https://mlsanity.com/paper/2508.04316/) (2025)
- [LiME: Lightweight Mixture of Experts for Efficient Multimodal Multi-task Learning](https://mlsanity.com/paper/2604.02338/) (2026)
- [LoCA: Spatially-Aware Low-Rank Convolutional Adaptation of Vision Foundation Models](https://mlsanity.com/paper/2607.06918/) (2026)

## Abstract (arXiv)

Large models such as Large Language Models (LLMs) and Vision Language Models (VLMs) have transformed artificial intelligence, powering applications in natural language processing, computer vision, and multimodal learning. However, fully fine-tuning these models remains expensive, requiring extensive computational resources, memory, and task-specific data. Parameter-Efficient Fine-Tuning (PEFT) has emerged as a promising solution that allows adapting large models to downstream tasks by updating only a small portion of parameters. This survey presents a comprehensive overview of PEFT techniques, focusing on their motivations, design principles, and effectiveness. We begin by analyzing the resource and accessibility challenges posed by traditional fine-tuning and highlight key issues, such as overfitting, catastrophic forgetting, and parameter inefficiency. We then introduce a structured taxonomy of PEFT methods -- grouped into additive, selective, reparameterized, hybrid, and unified frameworks -- and systematically compare their mechanisms and trade-offs. Beyond taxonomy, we explore the impact of PEFT across diverse domains, including language, vision, and generative modeling, showing how these techniques offer strong performance with lower resource costs. We also discuss important open challenges in scalability, interpretability, and robustness, and suggest future directions such as federated learning, domain adaptation, and theoretical grounding. Our goal is to provide a unified understanding of PEFT and its growing role in enabling practical, efficient, and sustainable use of large models.