mlsanityData to 7 Oct 2026

Understanding the Learning Dynamics of LoRA: A Gradient Flow Perspective on Low-Rank Adaptation in Matrix Factorization

arXiv 2503.06982 ↗First posted 10 Mar 2025cs.LG cs.AIcited by 13

Not on the map. Why?

Abstract (arXiv)

Despite the empirical success of Low-Rank Adaptation (LoRA) in fine-tuning pre-trained models, there is little theoretical understanding of how first-order methods with carefully crafted initialization adapt models to new tasks. In this work, we take the first step towards bridging this gap by theoretically analyzing the learning dynamics of LoRA for matrix factorization (MF) under gradient flow (GF), emphasizing the crucial role of initialization. For small initialization, we theoretically show that GF converges to a neighborhood of the optimal solution, with smaller initialization leading to lower final error. Our analysis shows that the final error is affected by the misalignment between the singular spaces of the pre-trained model and the target matrix, and reducing the initialization scale improves alignment. To address this misalignment, we propose a spectral initialization for LoRA in MF and theoretically prove that GF with small spectral initialization converges to the fine-tuning task with arbitrary precision. Numerical experiments from MF and image classification validate our findings.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 26 of its 55 references that are in the map. First posted on arXiv 10 Mar 2025.

Builds on

26 of 55 references.

References · 26

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2014How transferable are features in deep neural networks?cited by 1,216
  5. 2021Scaling Vision Transformerscited by 993
  6. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  7. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 365
  8. 2023Sentiment Analysis in the Era of Large Language Models: A Reality Checkcited by 262
  9. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 261
  10. 2019To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Taskscited by 167
  11. 2018Nonconvex Optimization Meets Low-Rank Matrix Factorization: An Overviewcited by 159
  12. 2022A Kernel-Based View of Language Model Fine-Tuningcited by 118
14 more
  1. 2020Spectral Methods for Data Science: A Statistical Perspectivecited by 111
  2. 2023The Expressive Power of Low-Rank Adaptationcited by 100
  3. 2024MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuningcited by 89
  4. 2021Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstructioncited by 77
  5. 2022What Makes Transfer Learning Work For Medical Images: Feature Reuse & Other Factorscited by 61
  6. 2023Saddle-to-Saddle Dynamics in Diagonal Linear Networkscited by 54
  7. 2024The Impact of Initialization on LoRA Finetuning Dynamicscited by 53
  8. 2024LoRA Training in the NTK Regime has No Spurious Local Minimacited by 48
  9. 2023Understanding Incremental Learning of Gradient Descent: A Fine-grained Analysis of Matrix Sensingcited by 40
  10. 2023Early Neuron Alignment in Two-layer ReLU Networks with Small Initializationcited by 28
  11. 2018Nonconvex Matrix Factorization from Rank-One Measurementscited by 13
  12. 2024NoRA: Nested Low-Rank Adaptation for Efficient Fine-Tuning Large Modelscited by 13
  13. 2013Spectral Compressed Sensing via Structured Matrix Completioncited by 7
  14. 2021Over-Parametrized Matrix Factorization in the Presence of Spurious Stationary Pointscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 10.

13 papers.

Sort:

By area:

  1. 2025On the Convergence Rate of LoRA Gradient Descentcited by 9
  2. 2026Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuningcited by 9
  3. 2026High-Dimensional Theory of LoRA Fine-Tuning in a Solvable Attention Modelcited by 3
  4. 2025Neural Collapse under Gradient Flow on Shallow ReLU Networks for Orthogonally Separable Datacited by 2
  5. 2026When pre-training hurts LoRA fine-tuning: a dynamical analysis via single-index modelscited by 2
  6. 2026Balanced LoRA: Removing Parameter Invariance to Accelerate Convergencecited by 2
  7. 2025Understanding Incremental Learning with Closed-form Solution to Gradient Flow on Overparamerterized Matrix Factorizationcited by 1
  8. 2026On the Convergence of Stochastic Low-Rank Adaptationcited by 1
  9. 2026ODELoRA: Training Low-Rank Adaptation by Solving Ordinary Differential Equationscited by 0
  10. 2026Dictionary-Aligned Concept Control for Safeguarding Multimodal LLMscited by 0
  11. 2026The Hidden Power of Scaling Factor in LoRA Optimizationcited by 0
  12. 2026ST-LoRA: Single Trajectory LoRA Ensemble for Uncertainty Aware Agricultural Segmentationcited by 0
1 more
  1. 2026A Dynamical Theory of LoRA in Continual Learningcited by 0