# Parameter Efficient Fine Tuning: A Comprehensive Analysis Across Applications

arXiv: https://arxiv.org/abs/2404.13506 · HTML: https://mlsanity.com/paper/2404.13506/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 52 references that are in the map.
- First posted on arXiv 21 Apr 2024.

## Builds on (most cited first, 30 of 35)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [Florence: A New Foundation Model for Computer Vision](https://mlsanity.com/paper/2111.11432/) (2021)
- [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://mlsanity.com/paper/2403.14608/) (2024)
- [Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling](https://mlsanity.com/paper/2111.14819/) (2021)
- [A Comprehensive Overview of Large Language Models](https://mlsanity.com/paper/2307.06435/) (2023)
- [FILIP: Fine-grained Interactive Language-Image Pre-Training](https://mlsanity.com/paper/2111.07783/) (2021)
- [Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data](https://mlsanity.com/paper/1908.04616/) (2019)
- [Understanding the Effects of RLHF on LLM Generalisation and Diversity](https://mlsanity.com/paper/2310.06452/) (2023)
- [LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models](https://mlsanity.com/paper/2304.01933/) (2023)
- [Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training](https://mlsanity.com/paper/2205.14401/) (2022)
- [Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgetting](https://mlsanity.com/paper/2004.12651/) (2020)
- [Full Parameter Fine-tuning for Large Language Models with Limited Resources](https://mlsanity.com/paper/2306.09782/) (2023)
- [End-to-End Speech Recognition: A Survey](https://mlsanity.com/paper/2303.03329/) (2023)
- [Parameter-efficient Model Adaptation for Vision Transformers](https://mlsanity.com/paper/2203.16329/) (2022)
- [Parameter-Efficient Fine-Tuning for Medical Image Analysis: The Missed Opportunity](https://mlsanity.com/paper/2305.08252/) (2023)
- [Exploring Wav2vec 2.0 fine-tuning for improved speech emotion recognition](https://mlsanity.com/paper/2110.06309/) (2021)
- [LLaMA-Reviewer: Advancing Code Review Automation with Large Language Models through Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2308.11148/) (2023)
- [Point-PEFT: Parameter-Efficient Fine-Tuning for 3D Pre-trained Models](https://mlsanity.com/paper/2310.03059/) (2023)
- [Controlling Conditional Language Models without Catastrophic Forgetting](https://mlsanity.com/paper/2112.00791/) (2021)
- [FairTune: Optimizing Parameter Efficient Fine Tuning for Fairness in Medical Image Analysis](https://mlsanity.com/paper/2310.05055/) (2023)
- [Faster and Lighter LLMs: A Survey on Current Challenges and Way Forward](https://mlsanity.com/paper/2402.01799/) (2024)
- [Sparsely Shared LoRA on Whisper for Child Speech Recognition](https://mlsanity.com/paper/2309.11756/) (2023)

## Built on by (most cited first, 25 of 25)

- [FlowerTune: A Cross-Domain Benchmark for Federated Fine-Tuning of Large Language Models](https://mlsanity.com/paper/2506.02961/) (2025)
- [Beyond LoRA: Exploring Efficient Fine-Tuning Techniques for Time Series Foundational Models](https://mlsanity.com/paper/2409.11302/) (2024)
- [IndicMMLU-Pro: Benchmarking Indic Large Language Models on Multi-Task Language Understanding](https://mlsanity.com/paper/2501.15747/) (2025)
- [Re-Imagining Multimodal Instruction Tuning: A Representation View](https://mlsanity.com/paper/2503.00723/) (2025)
- [Evolution of meta's llama models and parameter-efficient fine-tuning of large language models: a survey](https://mlsanity.com/paper/2510.12178/) (2025)
- [Decoding the Diversity: A Review of the Indic AI Research Landscape](https://mlsanity.com/paper/2406.09559/) (2024)
- [Parameter-Efficient Fine-Tuning of Large Language Models for Unit Test Generation: An Empirical Study](https://mlsanity.com/paper/2411.02462/) (2024)
- [KnowledgeSmith: Uncovering Knowledge Updating in LLMs with Model Editing and Unlearning](https://mlsanity.com/paper/2510.02392/) (2025)
- [LangSAMP: Language-Script Aware Multilingual Pretraining](https://mlsanity.com/paper/2409.18199/) (2024)
- [Efficient and Private: Memorisation under differentially private parameter-efficient fine-tuning in language models](https://mlsanity.com/paper/2411.15831/) (2024)
- [Unsupervised Human Preference Learning](https://mlsanity.com/paper/2410.03731/) (2024)
- [Parameter Efficient Fine-Tuning of Segment Anything Model for Biomedical Imaging](https://mlsanity.com/paper/2502.00418/) (2025)
- [Adaptive Budget Allocation for Orthogonal-Subspace Adapter Tuning in LLMs Continual Learning](https://mlsanity.com/paper/2505.22358/) (2025)
- [4,500 Seconds: Small Data Training Approaches for Deep UAV Audio Classification](https://mlsanity.com/paper/2505.23782/) (2025)
- [zFLoRA: Zero-Latency Fused Low-Rank Adapters](https://mlsanity.com/paper/2510.25784/) (2025)
- [MedPEFT-CL: Dual-Phase Parameter-Efficient Continual Learning with Medical Semantic Adapter and Bidirectional Memory Consolidation](https://mlsanity.com/paper/2511.17668/) (2025)
- [Shapelets-Enriched Selective Forecasting using Time Series Foundation Models](https://mlsanity.com/paper/2601.11821/) (2026)
- [Prompt Mechanisms in Medical Imaging: A Comprehensive Survey](https://mlsanity.com/paper/2507.01055/) (2025)
- [FLoRA: Fused forward-backward adapters for parameter efficient fine-tuning and reducing inference-time latencies of LLMs](https://mlsanity.com/paper/2511.00050/) (2025)
- [Loquetier: A Virtualized Multi-LoRA Framework for Unified LLM Fine-tuning and Serving](https://mlsanity.com/paper/2511.00101/) (2025)
- [Textual Data Bias Detection and Mitigation -- An Extensible Pipeline with Experimental Evaluation](https://mlsanity.com/paper/2512.10734/) (2025)
- [NanoNet: Parameter-Efficient Learning with Label-Scarce Supervision for Lightweight Text Mining Model](https://mlsanity.com/paper/2602.06093/) (2026)
- [Direct Preference Optimization for Chatbot Fine-Tuning: An Empirical Study](https://mlsanity.com/paper/2606.12881/) (2026)
- [Energy- and Memory-Efficient PEFT Methods for Personalized On-Device SLMs on Consumer GPUs](https://mlsanity.com/paper/2608.04488/) (2026)
- [From Manuals to Maintenance: Fine-Tuning MedGemma for Multi-Modal Imaging System Support in Low-Resource Settings](https://mlsanity.com/paper/2608.08896/) (2026)

## Abstract (arXiv)

The rise of deep learning has marked significant progress in fields such as computer vision, natural language processing, and medical imaging, primarily through the adaptation of pre-trained models for specific tasks. Traditional fine-tuning methods, involving adjustments to all parameters, face challenges due to high computational and memory demands. This has led to the development of Parameter Efficient Fine-Tuning (PEFT) techniques, which selectively update parameters to balance computational efficiency with performance. This review examines PEFT approaches, offering a detailed comparison of various strategies highlighting applications across different domains, including text generation, medical imaging, protein modeling, and speech synthesis. By assessing the effectiveness of PEFT methods in reducing computational load, speeding up training, and lowering memory usage, this paper contributes to making deep learning more accessible and adaptable, facilitating its wider application and encouraging innovation in model optimization. Ultimately, the paper aims to contribute towards insights into PEFT's evolving landscape, guiding researchers and practitioners in overcoming the limitations of conventional fine-tuning approaches.