mlsanityData to 7 Oct 2026

LoRA Training Provably Converges to a Low-Rank Global Minimum or It Fails Loudly (But it Probably Won't Fail)

arXiv 2502.09376 ↗First posted 13 Feb 2025cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

Low-rank adaptation (LoRA) has become a standard approach for fine-tuning large foundation models. However, our theoretical understanding of LoRA remains limited as prior analyses of LoRA's training dynamics either rely on linearization arguments or consider highly simplified setups. In this work, we analyze the LoRA loss landscape without such restrictive assumptions. We define two regimes: a "special regime", which includes idealized setups where linearization arguments hold, and a "generic regime" representing more realistic setups where linearization arguments do not hold. In the generic regime, we show that LoRA training converges to a global minimizer with low rank and small magnitude, or a qualitatively distinct solution with high rank and large magnitude. Finally, we argue that the zero-initialization and weight decay in LoRA training induce an implicit bias toward the low-rank, small-magnitude region of the parameter space -- where global minima lie -- thus shedding light on why LoRA training usually succeeds in finding global minima.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 28 of its 45 references that are in the map. First posted on arXiv 13 Feb 2025.

Builds on

28 of 45 references.

References · 28

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  7. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  8. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  9. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 365
  10. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  11. 2015Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decompositioncited by 241
  12. 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 231
16 more
  1. 2017No Spurious Local Minima in Nonconvex Low Rank Problems: A Unified Geometric Analysiscited by 127
  2. 2022A Kernel-Based View of Language Model Fine-Tuningcited by 118
  3. 2023The Expressive Power of Low-Rank Adaptationcited by 100
  4. 2016Global Optimality of Local Search for Low Rank Matrix Recoverycited by 81
  5. 2024LoRA-Pro: Are Low-Rank Adapters Properly Optimized?cited by 70
  6. 2024LoRA Training in the NTK Regime has No Spurious Local Minimacited by 48
  7. 2016Non-square matrix sensing without spurious local minima via the Burer-Monteiro approachcited by 38
  8. 2017Global Optimality in Low-rank Matrix Optimizationcited by 35
  9. 2023Implicit bias of SGD in L2-regularized linear DNNs: One-way jumps from high to low rankcited by 28
  10. 2024Weight decay induces low-rank attention layerscited by 27
  11. 2023Over-Parameterization Exponentially Slows Down Gradient Descent for Learning a Single Neuroncited by 25
  12. 2024Understanding Linear Probing then Fine-tuning Language Models from NTK Perspectivecited by 21
  13. 2023How Over-Parameterization Slows Down Gradient Descent in Matrix Sensing: The Curses of Symmetry and Initializationcited by 18
  14. 2022SGD and Weight Decay Secretly Minimize the Rank of Your Neural Networkcited by 17
  15. 2022Improved Global Guarantees for the Nonconvex Burer--Monteiro Factorization via Rank Overparameterizationcited by 13
  16. 2024Gradient dynamics for low-rank fine-tuning beyond kernelscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 15.

19 papers.

Sort:

By area:

  1. 2025On the Convergence Rate of LoRA Gradient Descentcited by 9
  2. 2026Why LoRA Fails to Forget: Regularized Low-Rank Adaptation Against Backdoors in Language Modelscited by 3
  3. 2026High-Dimensional Theory of LoRA Fine-Tuning in a Solvable Attention Modelcited by 3
  4. 2025Neural Collapse under Gradient Flow on Shallow ReLU Networks for Orthogonally Separable Datacited by 2
  5. 2026When pre-training hurts LoRA fine-tuning: a dynamical analysis via single-index modelscited by 2
  6. 2026Robustness of Agentic AI Systems via Adversarially-Aligned Jacobian Regularizationcited by 1
  7. 2026Curvature-Guided LoRA: Matching Full Fine-Tuning in Function Spacecited by 1
  8. 2026Convergent Stochastic Training of Multi-Headed Attention and Understanding LoRAcited by 1
  9. 2026LoRA vs. Full Fine-Tuning: A Theoretical Perspectivecited by 1
  10. 2026Beyond LoRA: Is Sparsity-Induced Adaptation Better?cited by 1
  11. 2026On the Convergence of Stochastic Low-Rank Adaptationcited by 1
  12. 2026SeFoRA: Sketch-Aggregated Federated Low-Rank Adaptation with Heterogeneous Client Rankscited by 1
7 more
  1. 2025Automatic Rank Determination for Low-Rank Adaptation via Submodular Function Maximizationcited by 0
  2. 2025LightSAE: Parameter-Efficient and Heterogeneity-Aware Embedding for IoT Multivariate Time Series Forecastingcited by 0
  3. 2026ODELoRA: Training Low-Rank Adaptation by Solving Ordinary Differential Equationscited by 0
  4. 2026AdaPaD: Adaptive Parallel Deflation for PEFT with Self-Correcting Rank Discoverycited by 0
  5. 2026ST-LoRA: Single Trajectory LoRA Ensemble for Uncertainty Aware Agricultural Segmentationcited by 0
  6. 2026Two-Timescale Fine-tuning Provably Learns New Features for Two-Layer ReLU Networkscited by 0
  7. 2026A Dynamical Theory of LoRA in Continual Learningcited by 0