mlsanityData to 14 Oct 2026

A Survey on LoRA of Large Language Models

arXiv 2407.11046 ↗First posted 8 Jul 2024cs.LG cs.AI cs.CLcited by 67

Not on the map. Why?

Abstract (arXiv)

Low-Rank Adaptation~(LoRA), which updates the dense neural network layers with pluggable low-rank matrices, is one of the best performed parameter efficient fine-tuning paradigms. Furthermore, it has significant advantages in cross-task generalization and privacy-preserving. Hence, LoRA has gained much attention recently, and the number of related literature demonstrates exponential growth. It is necessary to conduct a comprehensive overview of the current progress on LoRA. This survey categorizes and reviews the progress from the perspectives of (1) downstream adaptation improving variants that improve LoRA's performance on downstream tasks; (2) cross-task generalization methods that mix multiple LoRA plugins to achieve cross-task generalization; (3) efficiency-improving methods that boost the computation-efficiency of LoRA; (4) data privacy-preserving methods that use LoRA in federated learning; (5) application. Besides, this survey also discusses the future directions in this field. At last, we provide a Github page~footnotehrefhttps://github.com/ZJU-LLMs/Awesome-LoRAs.githttps://github.com/ZJU-LLMs/Awesome-LoRAs.git for readers to check the updates and initiate discussions on this survey paper.

Key facts. Cited by 67 papers in the map (data through 14 Oct 2026). Builds on 199 of its 228 references that are in the map. First posted on arXiv 8 Jul 2024.

Builds on

199 of 228 references.

References · 199

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  10. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  11. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  12. 2023A Survey of Large Language Modelscited by 2,576
38 more
  1. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,433
  2. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  3. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  4. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  5. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  6. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
  7. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  8. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  9. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 460
  10. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 377
  11. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 365
  12. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 364
  13. 2024LoRA Learns Less and Forgets Lesscited by 328
  14. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  15. 2023Customized Segment Anything Model for Medical Image Segmentationcited by 280
  16. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 269
  17. 2023Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Modelscited by 261
  18. 2023Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translationcited by 253
  19. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 244
  20. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 233
  21. 2023LCM-LoRA: A Universal Stable-Diffusion Acceleration Modulecited by 211
  22. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 204
  23. 2024InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learningcited by 198
  24. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 196
  25. 2023A mathematical perspective on Transformerscited by 164
  26. 2023Heterogeneous Federated Learning: State-of-the-art and Research Challengescited by 161
  27. 2024Improving LoRA in Privacy-preserving Federated Learningcited by 161
  28. 2023Composing Parameter-Efficient Modules with Arithmetic Operationscited by 156
  29. 2023QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Modelscited by 155
  30. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 152
  31. 2023The emergence of clusters in self-attention dynamicscited by 140
  32. 2021Sinkformers: Transformers with Doubly Stochastic Attentioncited by 138
  33. 2023LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuningcited by 137
  34. 2024Implicit Style-Content Separation using B-LoRAcited by 135
  35. 2022KronA: Parameter Efficient Tuning with Kronecker Adaptercited by 133
  36. 2023MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adaptercited by 128
  37. 2022A Kernel-Based View of Language Model Fine-Tuningcited by 121
  38. 2024Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Modelscited by 116

50 most cited of 199 All 199 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 41; 2026: 19.

67 papers.

Sort:

By area:

  1. 2024FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptationscited by 146
  2. 2025A Survey on the Optimization of Large Language Model-based Agentscited by 46
  3. 2025LLM Unlearning via Neural Activation Redirectioncited by 41
  4. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 37
  5. 2025PDE-Transformer: Efficient and Versatile Transformers for Physics Simulationscited by 32
  6. 2024Reassessing Layer Pruning in LLMs: New Insights and Methodscited by 24
  7. 2025CT-Agent: A Multimodal-LLM Agent for 3D CT Radiology Question Answeringcited by 21
  8. 2025The Rise of Small Language Models in Healthcare: A Comprehensive Surveycited by 18
  9. 2025Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Modelcited by 16
  10. 2025True Multimodal In-Context Learning Needs Attention to the Visual Contextcited by 16
  11. 2024NoRA: Nested Low-Rank Adaptation for Efficient Fine-Tuning Large Modelscited by 13
  12. 2025FinLoRA: Benchmarking LoRA Methods for Fine-Tuning LLMs on Financial Datasetscited by 12
38 more
  1. 2025Model Merging Scaling Laws in Large Language Modelscited by 12
  2. 2026Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuningcited by 10
  3. 2025Structure-Learnable Adapter Fine-Tuning for Parameter-Efficient Large Language Modelscited by 9
  4. 2025Graph-Aware Isomorphic Attention for Adaptive Dynamics in Transformerscited by 7
  5. 2025The Primacy of Magnitude in Low-Rank Adaptationcited by 7
  6. 2026TimeWarp: Evaluating Web Agents by Revisiting the Pastcited by 7
  7. 2024Adaptive Parameter-Efficient Federated Fine-Tuning on Heterogeneous Devicescited by 5
  8. 2025ElaLoRA: Elastic & Learnable Low-Rank Adaptation for Efficient Model Fine-Tuningcited by 5
  9. 2025scAgent: Universal Single-Cell Annotation via a LLM Agentcited by 5
  10. 2025The Panaceas for Improving Low-Rank Decomposition in Communication-Efficient Federated Learningcited by 5
  11. 2025Does Low Rank Adaptation Lead to Lower Robustness against Training-Time Attacks?cited by 4
  12. 2025HiLoRA: Adaptive Hierarchical LoRA Routing for Training-Free Domain Generalizationcited by 4
  13. 2024Rapid Adaptation of Earth Observation Foundation Models for Segmentationcited by 3
  14. 2025S'MoRE: Structural Mixture of Residual Experts for Parameter-Efficient LLM Fine-tuningcited by 3
  15. 2025FedHL: Federated Learning for Heterogeneous Low-Rank Adaptation via Unbiased Aggregationcited by 3
  16. 2025Structured Gradient Guidance for Few-Shot Adaptation in Large Language Modelscited by 3
  17. 2026Low-Rank Adaptation Redux for Large Modelscited by 3
  18. 2026Localized LoRA-MoE: Block-wise Low-Rank Experts With Adaptive Routingcited by 3
  19. 2025Hallucinations and Truth: A Comprehensive Accuracy Evaluation of RAG, LoRA and DoRAcited by 2
  20. 2025MetaLoRA: Tensor-Enhanced Adaptive Low-Rank Fine-tuningcited by 2
  21. 2025ConsNoTrainLoRA: Data-driven Weight Initialization of Low-rank Adapters using Constraintscited by 2
  22. 2025Modelling and Classifying the Components of a Literature Reviewcited by 2
  23. 2025Building Large-Scale English-Romanian Literary Translation Resources with Open Modelscited by 2
  24. 2026A Unified Study of LoRA Variants: Taxonomy, Review, Codebase, and Empirical Evaluationcited by 2
  25. 2026When pre-training hurts LoRA fine-tuning: a dynamical analysis via single-index modelscited by 2
  26. 2026A Language-Guided Bayesian Optimization for Efficient LoRA Hyperparameter Searchcited by 2
  27. 2025HRP: High-Rank Preheating for Superior LoRA Initializationcited by 1
  28. 2025Localized LoRA: A Structured Low-Rank Approximation for Efficient Fine-Tuningcited by 1
  29. 2025FedRPCA: Enhancing Federated LoRA Aggregation Using Robust PCAcited by 1
  30. 2025DMFI: A Dual-Modality Log Analysis Framework for Insider Threat Detection with LoRA-Tuned Language Modelscited by 1
  31. 2025Hypernetwork-Driven Low-Rank Adaptation Across Attention Headscited by 1
  32. 2025Domain-Adapted Pre-trained Language Models for Implicit Information Extraction in Crash Narrativescited by 1
  33. 2024Low-rank Adaptation-based All-Weather Removal for Autonomous Navigationcited by 0
  34. 2025Collaborative Editable Modelcited by 0
  35. 2025Automatic Rank Determination for Low-Rank Adaptation via Submodular Function Maximizationcited by 0
  36. 2025Exploring and Reshaping the Weight Distribution in LLMcited by 0
  37. 2025IPA: An Information-Reconstructive Input Projection Framework for Efficient Foundation Model Adaptationcited by 0
  38. 2025A cybersecurity AI agent selection and decision support frameworkcited by 0

50 most cited of 67 All 67 (JSON)