mlsanityData to 14 Oct 2026

On the Convergence Rate of LoRA Gradient Descent

arXiv 2512.18248 ↗First posted 20 Dec 2025cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

The low-rank adaptation (LoRA) algorithm for fine-tuning large models has grown popular in recent years due to its remarkable performance and low computational requirements. LoRA trains two ``adapter" matrices that form a low-rank representation of the model parameters, thereby massively reducing the number of parameters that need to be updated at every step. Although LoRA is simple, its convergence is poorly understood due to the lack of Lipschitz smoothness, a key condition for classic convergence analyses. As a result, current theoretical results only consider asymptotic behavior or assume strong boundedness conditions which artificially enforce Lipschitz smoothness. In this work, we provide for the first time a non-asymptotic convergence analysis of the textitoriginal LoRA gradient descent algorithm, which reflects widespread practice, without such assumptions. Our work relies on three key steps: i) reformulating the problem in terms of the outer product of the stacked adapter matrices, ii) a modified descent lemma for the ``Lipschitz-like" reparametrized function, and iii) controlling the step size. With this approach, we prove that LoRA gradient descent converges to a stationary point at rate O(frac1log T), where T is the number of iterations. We conduct numerical experiments to validate our theoretical findings.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 35 of its 42 references that are in the map. First posted on arXiv 20 Dec 2025.

Builds on

35 of 42 references.

References · 35

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  4. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
  5. 2015Cyclical Learning Rates for Training Neural Networkscited by 629
  6. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  7. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 377
  8. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 370
  9. 2024LoRA Learns Less and Forgets Lesscited by 328
  10. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 269
  11. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 204
  12. 2023Handbook of Convergence Theorems for (Stochastic) Gradient Methodscited by 163
23 more
  1. 2024Improving LoRA in Privacy-preserving Federated Learningcited by 161
  2. 2024LoRA-GA: Low-Rank Adaptation with Gradient Approximationcited by 141
  3. 2022A Kernel-Based View of Language Model Fine-Tuningcited by 121
  4. 2024LoRA vs Full Fine-tuning: An Illusion of Equivalencecited by 109
  5. 2023The Expressive Power of Low-Rank Adaptationcited by 102
  6. 2019Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobiancited by 67
  7. 2024Riemannian Preconditioned LoRA for Fine-Tuning Foundation Modelscited by 52
  8. 2024LoRA Training in the NTK Regime has No Spurious Local Minimacited by 48
  9. 2024OLoRA: Orthonormal Low-Rank Adaptation of Large Language Modelscited by 39
  10. 2024LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statisticscited by 38
  11. 2025LoRA-One: One-Step Full Gradient Could Suffice for Fine-Tuning Large Language Models, Provably and Efficientlycited by 38
  12. 2024Subspace Optimization for Large Language Models with Convergence Guaranteescited by 34
  13. 2024Compressible Dynamics in Deep Overparameterized Low-Rank Learning & Adaptationcited by 26
  14. 2024Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscapecited by 20
  15. 2025LoRA Training Provably Converges to a Low-Rank Global Minimum or It Fails Loudly (But it Probably Won't Fail)cited by 20
  16. 2025A Memory Efficient Randomized Subspace Optimization Method for Training Large Language Modelscited by 16
  17. 2025Decentralized Low-Rank Fine-Tuning of Large Language Modelscited by 14
  18. 2024Randomized Asymmetric Chain of LoRA: The First Meaningful Theoretical Framework for Low-Rank Adaptationcited by 13
  19. 2025Understanding the Learning Dynamics of LoRA: A Gradient Flow Perspective on Low-Rank Adaptation in Matrix Factorizationcited by 13
  20. 2025Federated Low-Rank Adaptation for Foundation Models: A Surveycited by 9
  21. 2024Communication-Efficient Federated Low-Rank Update Algorithm and its Connection to Implicit Regularizationcited by 6
  22. 2025Bernoulli-LoRA: A Theoretical Framework for Randomized Low-Rank Adaptationcited by 4
  23. 2025Kron-LoRA: Hybrid Kronecker-LoRA Adapters for Scalable, Sustainable Fine-tuningcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 10.

10 papers.

  1. 2026High-Dimensional Theory of LoRA Fine-Tuning in a Solvable Attention Modelcited by 3
  2. 2026When pre-training hurts LoRA fine-tuning: a dynamical analysis via single-index modelscited by 2
  3. 2026Beyond LoRA: Is Sparsity-Induced Adaptation Better?cited by 1
  4. 2026On the Convergence of Stochastic Low-Rank Adaptationcited by 1
  5. 2026MARS: Harmonizing Multimodal Convergence via Adaptive Rank Searchcited by 0
  6. 2026The Hidden Power of Scaling Factor in LoRA Optimizationcited by 0
  7. 2026Convergence Guarantees of Gradient Descent for Neural Networks via Generalized Lipschitz Smoothnesscited by 0
  8. 2026LoRA-TSD: Tangent-Space Spectral Descent for LoRA via Muon-Style Updatescited by 0
  9. 2026Reasoning-Preserving Fine-Tuning of Post-RL LLMs with Null-Basis LoRAcited by 0
  10. 2026Global Exponential Convergence of Two-Layer Linear Network Trainingcited by 0