mlsanityData to 14 Oct 2026

LoRA Training in the NTK Regime has No Spurious Local Minima

arXiv 2402.11867 ↗First posted 19 Feb 2024cs.LG math.OCcited by 48

Not on the map. Why?

Abstract (arXiv)

Low-rank adaptation (LoRA) has become the standard approach for parameter-efficient fine-tuning of large language models (LLM), but our theoretical understanding of LoRA has been limited. In this work, we theoretically analyze LoRA fine-tuning in the neural tangent kernel (NTK) regime with N data points, showing: (i) full fine-tuning (without LoRA) admits a low-rank solution of rank rlesssim √N; (ii) using LoRA with rank rgtrsim √N eliminates spurious local minima, allowing gradient descent to find the low-rank solutions; (iii) the low-rank solution found using LoRA generalizes well.

Key facts. Cited by 48 papers in the map (data through 14 Oct 2026). Builds on 45 of its 76 references that are in the map. First posted on arXiv 19 Feb 2024.

Builds on

45 of 76 references.

References · 45

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  5. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  6. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  7. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  8. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 938
  9. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  10. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 724
  11. 2017Spectrally-normalized margin bounds for neural networkscited by 686
  12. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
33 more
  1. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  2. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
  3. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  4. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 552
  5. 2017Sharp Minima Can Generalize For Deep Netscited by 466
  6. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 406
  7. 2017The Expressive Power of Neural Networks: A View from the Widthcited by 320
  8. 2017How to Escape Saddle Points Efficientlycited by 285
  9. 2015Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decompositioncited by 243
  10. 2023Looped Transformers as Programmable Computerscited by 237
  11. 2020Understanding the Difficulty of Training Transformerscited by 210
  12. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 204
  13. 2016A vector-contraction inequality for Rademacher complexitiescited by 142
  14. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 141
  15. 2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 141
  16. 2023Continual Diffusion: Continual Customization of Text-to-Image Diffusion with C-LoRAcited by 141
  17. 2017Theoretical insights into the optimization landscape of over-parameterized shallow neural networkscited by 128
  18. 2016Matrix Completion has No Spurious Local Minimumcited by 125
  19. 2022A Kernel-Based View of Language Model Fine-Tuningcited by 121
  20. 2023Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluationcited by 109
  21. 2018On the Power of Over-parametrization in Neural Networks with Quadratic Activationcited by 102
  22. 2021Statistically Meaningful Approximation: a Case Study on Approximating Turing Machines with Transformerscited by 102
  23. 2023The Expressive Power of Low-Rank Adaptationcited by 102
  24. 2020Neural Networks are Convex Regularizers: Exact Polynomial-time Convex Optimization Formulations for Two-layer Networkscited by 98
  25. 2022More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalizecited by 91
  26. 2017Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapescited by 84
  27. 2018On the Convergence Rate of Training Recurrent Neural Networkscited by 82
  28. 2017Gradient Descent Can Take Exponential Time to Escape Saddle Pointscited by 74
  29. 2023Transformers learn through gradual rank increasecited by 52
  30. 2016The non-convex Burer-Monteiro approach works on smooth semidefinite programscited by 46
  31. 2023Understanding Incremental Learning of Gradient Descent: A Fine-grained Analysis of Matrix Sensingcited by 42
  32. 2023Minimum Width of Leaky-ReLU Neural Networks for Uniform Universal Approximationcited by 13
  33. 2018Smoothed analysis for low-rank solutions to semidefinite programs in quadratic penalty formcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 10; 2025: 16; 2026: 22.

48 papers.

Sort:

By area:

  1. 2024A Survey on LoRA of Large Language Modelscited by 67
  2. 2025Training Dynamics of In-Context Learning in Linear Attentioncited by 46
  3. 2025When is Task Vector Provably Effective for Model Editing? A Generalization Analysis of Nonlinear Transformerscited by 38
  4. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 37
  5. 2025LoRA Training Provably Converges to a Low-Rank Global Minimum or It Fails Loudly (But it Probably Won't Fail)cited by 20
  6. 2024On the Crucial Role of Initialization for Matrix Factorizationcited by 15
  7. 2024Propulsion: Steering LLM with Tiny Fine-Tuningcited by 14
  8. 2025Decentralized Low-Rank Fine-Tuning of Large Language Modelscited by 14
  9. 2025Understanding the Learning Dynamics of LoRA: A Gradient Flow Perspective on Low-Rank Adaptation in Matrix Factorizationcited by 13
  10. 2024Parameter-Efficient Fine-Tuning of State Space Modelscited by 11
  11. 2025Sharp Generalization Bounds for Foundation Models with Asymmetric Randomized Low-Rank Adapterscited by 11
  12. 2025Merge before Forget: A Single LoRA Continual Learning via Continual Mergingcited by 11
36 more
  1. 2025On the Convergence Rate of LoRA Gradient Descentcited by 10
  2. 2024AutoLoRA: AutoGuidance Meets Low-Rank Adaptation for Diffusion Modelscited by 7
  3. 2025LoRE-Merging: Exploring Low-Rank Estimation For Large Language Model Mergingcited by 7
  4. 2025The Primacy of Magnitude in Low-Rank Adaptationcited by 7
  5. 2024Gradient dynamics for low-rank fine-tuning beyond kernelscited by 5
  6. 2024On Generalisation Error Bounds for Transformerscited by 4
  7. 2025Does Low Rank Adaptation Lead to Lower Robustness against Training-Time Attacks?cited by 4
  8. 2025FedHL: Federated Learning for Heterogeneous Low-Rank Adaptation via Unbiased Aggregationcited by 3
  9. 2026Why LoRA Resists Label Noise: A Theoretical Framework for Noise-Robust Parameter-Efficient Fine-Tuningcited by 3
  10. 2026Low-Rank Adaptation Redux for Large Modelscited by 3
  11. 2026Balanced LoRA: Removing Parameter Invariance to Accelerate Convergencecited by 3
  12. 2026High-Dimensional Theory of LoRA Fine-Tuning in a Solvable Attention Modelcited by 3
  13. 2024ALLoRA: Adaptive Learning Rate Mitigates LoRA Fatal Flawscited by 2
  14. 2024Provable Meta-Learning with Low-Rank Adaptationscited by 2
  15. 2025Neural Collapse under Gradient Flow on Shallow ReLU Networks for Orthogonally Separable Datacited by 2
  16. 2026Artificial Entanglement in the Fine-Tuning of Large Language Modelscited by 2
  17. 2026When pre-training hurts LoRA fine-tuning: a dynamical analysis via single-index modelscited by 2
  18. 2026Linearization Explains Fine-Tuning in Large Language Modelscited by 2
  19. 2026Tight Sample Complexity for Low-Rank Adaptation: Matching Bounds and Rank Selectioncited by 2
  20. 2025Safety Alignment Depth in Large Language Models: A Markov Chain Perspectivecited by 1
  21. 2026When Is Compositional Reasoning Learnable from Verifiable Rewards?cited by 1
  22. 2026Sharpness-Aware Minimization in Logit Space Efficiently Enhances Direct Preference Optimizationcited by 1
  23. 2026Curvature-Guided LoRA: Matching Full Fine-Tuning in Function Spacecited by 1
  24. 2026Rethinking the Rank Threshold for LoRA Fine-Tuningcited by 1
  25. 2026Convergent Stochastic Training of Multi-Headed Attention and Understanding LoRAcited by 1
  26. 2026On the Convergence of Stochastic Low-Rank Adaptationcited by 1
  27. 2026When Do Task Vectors Interfere? Mapping the Validity Boundaries of Weight-Space Compositioncited by 1
  28. 2025Norm-Bounded Low-Rank Adaptationcited by 0
  29. 2025Automatic Rank Determination for Low-Rank Adaptation via Submodular Function Maximizationcited by 0
  30. 2026ODELoRA: Training Low-Rank Adaptation by Solving Ordinary Differential Equationscited by 0
  31. 2026LOFT: Low-Rank Orthogonal Fine-Tuning via Task-Aware Support Selectioncited by 0
  32. 2026Not How Many, But Which: Parameter Placement in Low-Rank Adaptationcited by 0
  33. 2026The Hidden Power of Scaling Factor in LoRA Optimizationcited by 0
  34. 2026Dive Into the Implicit Biases of Low-rank Vision-language Alignmentcited by 0
  35. 2026Certification Frontiers for Gaussian LoRA: Independent Priors, Posterior Risk, and Prediction-Preserving Balancingcited by 0
  36. 2026Two-Timescale Fine-tuning Provably Learns New Features for Two-Layer ReLU Networkscited by 0