# A Survey on LoRA of Large Language Models

arXiv: https://arxiv.org/abs/2407.11046 · HTML: https://mlsanity.com/paper/2407.11046/

## Key facts

- Cited by 67 papers in the map (data through 14 Oct 2026).
- Builds on 199 of its 228 references that are in the map.
- First posted on arXiv 8 Jul 2024.

## Builds on (most cited first, 30 of 199)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality](https://mlsanity.com/paper/2304.14178/) (2023)
- [Towards a Unified View of Parameter-Efficient Transfer Learning](https://mlsanity.com/paper/2110.04366/) (2021)
- [DoRA: Weight-Decomposed Low-Rank Adaptation](https://mlsanity.com/paper/2402.09353/) (2024)
- [Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning](https://mlsanity.com/paper/2012.13255/) (2020)
- [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://mlsanity.com/paper/2403.14608/) (2024)
- [GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection](https://mlsanity.com/paper/2403.03507/) (2024)
- [Compacter: Efficient Low-Rank Hypercomplex Adapter Layers](https://mlsanity.com/paper/2106.04647/) (2021)
- [LoRA+: Efficient Low Rank Adaptation of Large Models](https://mlsanity.com/paper/2402.12354/) (2024)
- [Laplace Redux -- Effortless Bayesian Deep Learning](https://mlsanity.com/paper/2106.14806/) (2021)
- [LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition](https://mlsanity.com/paper/2307.13269/) (2023)
- [LoRA Learns Less and Forgets Less](https://mlsanity.com/paper/2405.09673/) (2024)
- [InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model](https://mlsanity.com/paper/2401.16420/) (2024)
- [Customized Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.13785/) (2023)
- [PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models](https://mlsanity.com/paper/2404.02948/) (2024)
- [Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models](https://mlsanity.com/paper/2305.18292/) (2023)
- [Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation](https://mlsanity.com/paper/2306.07954/) (2023)

## Built on by (most cited first, 30 of 67)

- [FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations](https://mlsanity.com/paper/2409.05976/) (2024)
- [A Survey on the Optimization of Large Language Model-based Agents](https://mlsanity.com/paper/2503.12434/) (2025)
- [LLM Unlearning via Neural Activation Redirection](https://mlsanity.com/paper/2502.07218/) (2025)
- [Low-Rank Adaptation for Foundation Models: A Comprehensive Review](https://mlsanity.com/paper/2501.00365/) (2024)
- [PDE-Transformer: Efficient and Versatile Transformers for Physics Simulations](https://mlsanity.com/paper/2505.24717/) (2025)
- [Reassessing Layer Pruning in LLMs: New Insights and Methods](https://mlsanity.com/paper/2411.15558/) (2024)
- [CT-Agent: A Multimodal-LLM Agent for 3D CT Radiology Question Answering](https://mlsanity.com/paper/2505.16229/) (2025)
- [The Rise of Small Language Models in Healthcare: A Comprehensive Survey](https://mlsanity.com/paper/2504.17119/) (2025)
- [Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Model](https://mlsanity.com/paper/2503.04543/) (2025)
- [True Multimodal In-Context Learning Needs Attention to the Visual Context](https://mlsanity.com/paper/2507.15807/) (2025)
- [NoRA: Nested Low-Rank Adaptation for Efficient Fine-Tuning Large Models](https://mlsanity.com/paper/2408.10280/) (2024)
- [FinLoRA: Benchmarking LoRA Methods for Fine-Tuning LLMs on Financial Datasets](https://mlsanity.com/paper/2505.19819/) (2025)
- [Model Merging Scaling Laws in Large Language Models](https://mlsanity.com/paper/2509.24244/) (2025)
- [Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning](https://mlsanity.com/paper/2602.04998/) (2026)
- [Structure-Learnable Adapter Fine-Tuning for Parameter-Efficient Large Language Models](https://mlsanity.com/paper/2509.03057/) (2025)
- [Graph-Aware Isomorphic Attention for Adaptive Dynamics in Transformers](https://mlsanity.com/paper/2501.02393/) (2025)
- [The Primacy of Magnitude in Low-Rank Adaptation](https://mlsanity.com/paper/2507.06558/) (2025)
- [TimeWarp: Evaluating Web Agents by Revisiting the Past](https://mlsanity.com/paper/2603.04949/) (2026)
- [Adaptive Parameter-Efficient Federated Fine-Tuning on Heterogeneous Devices](https://mlsanity.com/paper/2412.20004/) (2024)
- [ElaLoRA: Elastic & Learnable Low-Rank Adaptation for Efficient Model Fine-Tuning](https://mlsanity.com/paper/2504.00254/) (2025)
- [scAgent: Universal Single-Cell Annotation via a LLM Agent](https://mlsanity.com/paper/2504.04698/) (2025)
- [The Panaceas for Improving Low-Rank Decomposition in Communication-Efficient Federated Learning](https://mlsanity.com/paper/2505.23176/) (2025)
- [Does Low Rank Adaptation Lead to Lower Robustness against Training-Time Attacks?](https://mlsanity.com/paper/2505.12871/) (2025)
- [HiLoRA: Adaptive Hierarchical LoRA Routing for Training-Free Domain Generalization](https://mlsanity.com/paper/2510.12266/) (2025)
- [Rapid Adaptation of Earth Observation Foundation Models for Segmentation](https://mlsanity.com/paper/2409.09907/) (2024)
- [S'MoRE: Structural Mixture of Residual Experts for Parameter-Efficient LLM Fine-tuning](https://mlsanity.com/paper/2504.06426/) (2025)
- [FedHL: Federated Learning for Heterogeneous Low-Rank Adaptation via Unbiased Aggregation](https://mlsanity.com/paper/2505.18494/) (2025)
- [Structured Gradient Guidance for Few-Shot Adaptation in Large Language Models](https://mlsanity.com/paper/2506.00726/) (2025)
- [Low-Rank Adaptation Redux for Large Models](https://mlsanity.com/paper/2604.21905/) (2026)
- [Localized LoRA-MoE: Block-wise Low-Rank Experts With Adaptive Routing](https://mlsanity.com/paper/2607.05114/) (2026)

## Abstract (arXiv)

Low-Rank Adaptation~(LoRA), which updates the dense neural network layers with pluggable low-rank matrices, is one of the best performed parameter efficient fine-tuning paradigms. Furthermore, it has significant advantages in cross-task generalization and privacy-preserving. Hence, LoRA has gained much attention recently, and the number of related literature demonstrates exponential growth. It is necessary to conduct a comprehensive overview of the current progress on LoRA. This survey categorizes and reviews the progress from the perspectives of (1) downstream adaptation improving variants that improve LoRA's performance on downstream tasks; (2) cross-task generalization methods that mix multiple LoRA plugins to achieve cross-task generalization; (3) efficiency-improving methods that boost the computation-efficiency of LoRA; (4) data privacy-preserving methods that use LoRA in federated learning; (5) application. Besides, this survey also discusses the future directions in this field. At last, we provide a Github page~footnotehrefhttps://github.com/ZJU-LLMs/Awesome-LoRAs.githttps://github.com/ZJU-LLMs/Awesome-LoRAs.git for readers to check the updates and initiate discussions on this survey paper.