mlsanityData to 14 Oct 2026

LoRA+: Efficient Low Rank Adaptation of Large Models

arXiv 2402.12354 ↗First posted 19 Feb 2024cs.LG cs.AI cs.CLcited by 377

Not on the map. Why?

Abstract (arXiv)

In this paper, we show that Low Rank Adaptation (LoRA) as originally introduced in Hu et al. (2021) leads to suboptimal finetuning of models with large width (embedding dimension). This is due to the fact that adapter matrices A and B in LoRA are updated with the same learning rate. Using scaling arguments for large width networks, we demonstrate that using the same learning rate for A and B does not allow efficient feature learning. We then show that this suboptimality of LoRA can be corrected simply by setting different learning rates for the LoRA adapter matrices A and B with a well-chosen ratio. We call this proposed algorithm LoRA+. In our extensive experiments, LoRA+ improves performance (1-2 % improvements) and finetuning speed (up to ~ 2X SpeedUp), at the same computational cost as LoRA.

Key facts. Cited by 377 papers in the map (data through 14 Oct 2026). Builds on 32 of its 38 references that are in the map. First posted on arXiv 19 Feb 2024.

Builds on

32 of 38 references.

References · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  11. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  12. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
20 more
  1. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  3. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  4. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  5. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  6. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 408
  7. 2023VeRA: Vector-based Random Matrix Adaptationcited by 300
  8. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 288
  9. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 196
  10. 2016Deep Information Propagationcited by 195
  11. 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 149
  12. 2019On the Impact of the Activation Function on Deep Neural Networks Trainingcited by 102
  13. 2023The Expressive Power of Low-Rank Adaptationcited by 102
  14. 2023Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networkscited by 97
  15. 2023Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limitcited by 66
  16. 2020Stable ResNetcited by 49
  17. 2023The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limitcited by 47
  18. 2023Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagationcited by 45
  19. 2023Tensor Programs IVb: Adaptive Optimization in the Infinite-Width Limitcited by 38
  20. 2022On the infinite-depth limit of finite-width neural networkscited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 93; 2025: 154; 2026: 130.

377 papers.

Sort:

By area:

  1. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  2. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  3. 2024LoRA Learns Less and Forgets Lesscited by 328
  4. 2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Expertscited by 147
  5. 2024LoRA-GA: Low-Rank Adaptation with Gradient Approximationcited by 141
  6. 2024LoRA vs Full Fine-tuning: An Illusion of Equivalencecited by 109
  7. 2024MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuningcited by 94
  8. 2024Selective Aggregation for Low-Rank Adaptation in Federated Learningcited by 80
  9. 2024LoRA-Pro: Are Low-Rank Adapters Properly Optimized?cited by 77
  10. 2024LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameterscited by 76
  11. 2024Tracking Meets LoRA: Faster Training, Larger Model, Stronger Performancecited by 70
  12. 2024A Survey on LoRA of Large Language Modelscited by 67
38 more
  1. 2024Learning to Route LLMs with Confidence Tokenscited by 67
  2. 2026MIRAGE: The Illusion of Visual Understandingcited by 65
  3. 2024The Impact of Initialization on LoRA Finetuning Dynamicscited by 54
  4. 2025K-LoRA: Unlocking Training-Free Fusion of Any Subject and Style LoRAscited by 48
  5. 2024Mixture-of-Subspaces in Low-Rank Adaptationcited by 47
  6. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 47
  7. 2024SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretrainingcited by 45
  8. 2024OLoRA: Orthonormal Low-Rank Adaptation of Large Language Modelscited by 39
  9. 2024Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradientscited by 38
  10. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 37
  11. 2025SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuningcited by 33
  12. 2024LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimizationcited by 32
  13. 2024MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuningcited by 31
  14. 2024Computational Limits of Low-Rank Adaptation (LoRA) Fine-Tuning for Transformer Modelscited by 31
  15. 2025HSplitLoRA: A Heterogeneous Split Parameter-Efficient Fine-Tuning Framework for Large Language Modelscited by 31
  16. 2024Compute Better Spent: Replacing Dense Layers with Structured Matricescited by 28
  17. 2024Hyperbolic Fine-Tuning for Large Language Modelscited by 27
  18. 2024From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applicationscited by 25
  19. 2024QuanTA: Efficient High-Rank Fine-Tuning of LLMs with Quantum-Informed Tensor Adaptationcited by 23
  20. 2025Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignmentcited by 23
  21. 2024Learning on LoRAs: GL-Equivariant Processing of Low-Rank Weight Spaces for Large Finetuned Modelscited by 22
  22. 2024Towards Robust and Efficient Federated Low-Rank Adaptation with Heterogeneous Clientscited by 22
  23. 2024Spectral Adapter: Fine-Tuning in Spectral Spacecited by 21
  24. 2025GSQ-Tuning: Group-Shared Exponents Integer in Fully Quantized Training for LLMs On-Device Fine-tuningcited by 21
  25. 2024Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscapecited by 20
  26. 2024Parameter Efficient Fine-tuning via Explained Variance Adaptationcited by 20
  27. 2025LoRA Training Provably Converges to a Low-Rank Global Minimum or It Fails Loudly (But it Probably Won't Fail)cited by 20
  28. 2025PoLAR: Polar-Decomposed Low-Rank Adapter Representationcited by 20
  29. 2025Drag-and-Drop LLMs: Zero-Shot Prompt-to-Weightscited by 20
  30. 2025LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptationcited by 19
  31. 2025LoRA-Mixer: Coordinate Modular LoRA Experts Through Serial Attention Routingcited by 18
  32. 2024Sparse High Rank Adapterscited by 17
  33. 2024MTL-LoRA: Low-Rank Adaptation for Multi-Task Learningcited by 17
  34. 2025Robust Federated Finetuning of LLMs via Alternating Optimization of LoRAcited by 17
  35. 2025Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Modelcited by 16
  36. 2025Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamicscited by 16
  37. 2024Advancing Multimodal Large Language Models with Quantization-Aware Scale Learning for Efficient Adaptationcited by 15
  38. 2024A Unified Framework with Multimodal Fine-tuning for Remote Sensing Semantic Segmentationcited by 15

50 most cited of 377 All 377 (JSON)