On the Convergence of Stochastic Low-Rank Adaptation
Not on the map. Why?
Abstract (arXiv)
Low-rank adaptation (LoRA) optimizes J(B,A)=mathcal L(Wbase+sBA) over two adapters B ∈ ℝm × r and A ∈ ℝr × n that form a low-rank update to a frozen pretrained weight matrix Wbase ∈ ℝm × n. The prior analysis shows LoRA-GD takes expO(ε-2) oracle calls to find an ε-stationary point such that |nabla J(B,A)|≤ ε in the deterministic setting. We sharpen the analysis and show that O(ε-4) full-gradient evaluations suffice for the same first-order criterion. We further study stochastic LoRA under unbiased gradient estimates and finite variance. We propose LoRA-NSGDM, which finds an ε-stationary point with O(ε-8) stochastic oracle complexity. Under the additional mean-square smoothness condition, we use variance reduction strategy and propose LoRA-STORM, which improves the stochastic oracle complexity to O(ε-6).
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 29 of its 34 references that are in the map. First posted on arXiv 24 Jul 2026.
Builds on
29 of 34 references.
References · 29
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
- 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
- 2024TinyLlama: An Open-Source Small Language Modelcited by 618
- 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 377
- 2024LoRA Learns Less and Forgets Lesscited by 328
- 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 306
- 2018SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path Integrated Differential Estimatorcited by 287
- 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 269
- 2017SARAH: A Novel Method for Machine Learning Problems Using Stochastic Recursive Gradientcited by 220
- 2024Improving LoRA in Privacy-preserving Federated Learningcited by 161
17 more
- 2016Stochastic Variance Reduction for Nonconvex Optimizationcited by 145
- 2024LoRA-GA: Low-Rank Adaptation with Gradient Approximationcited by 141
- 2020Momentum Improves Normalized SGDcited by 139
- 2020PAGE: A Simple and Optimal Probabilistic Gradient Estimator for Nonconvex Optimizationcited by 119
- 2024LoRA vs Full Fine-tuning: An Illusion of Equivalencecited by 109
- 2023The Expressive Power of Low-Rank Adaptationcited by 102
- 2024Riemannian Preconditioned LoRA for Fine-Tuning Foundation Modelscited by 52
- 2024LoRA Training in the NTK Regime has No Spurious Local Minimacited by 48
- 2025LoRA-One: One-Step Full Gradient Could Suffice for Fine-Tuning Large Language Models, Provably and Efficientlycited by 38
- 2019ProxSARAH: An Efficient Algorithmic Framework for Stochastic Composite Nonconvex Optimizationcited by 37
- 2025LoRA Training Provably Converges to a Low-Rank Global Minimum or It Fails Loudly (But it Probably Won't Fail)cited by 20
- 2020Momentum-based variance-reduced proximal stochastic gradient method for composite nonconvex stochastic optimizationcited by 14
- 2025Decentralized Low-Rank Fine-Tuning of Large Language Modelscited by 14
- 2024Randomized Asymmetric Chain of LoRA: The First Meaningful Theoretical Framework for Low-Rank Adaptationcited by 13
- 2025Understanding the Learning Dynamics of LoRA: A Gradient Flow Perspective on Low-Rank Adaptation in Matrix Factorizationcited by 13
- 2025On the Convergence Rate of LoRA Gradient Descentcited by 10
- 2025Bernoulli-LoRA: A Theoretical Framework for Randomized Low-Rank Adaptationcited by 4
Built on by
Citing papers per year: 2026: 1.
1 paper.