# ODELoRA: Training Low-Rank Adaptation by Solving Ordinary Differential Equations

arXiv: https://arxiv.org/abs/2602.07479 · HTML: https://mlsanity.com/paper/2602.07479/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 35 of its 55 references that are in the map.
- First posted on arXiv 7 Feb 2026.

## Builds on (most cited first, 30 of 35)

- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [DoRA: Weight-Decomposed Low-Rank Adaptation](https://mlsanity.com/paper/2402.09353/) (2024)
- [Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent](https://mlsanity.com/paper/1902.06720/) (2019)
- [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://mlsanity.com/paper/2403.14608/) (2024)
- [LoRA+: Efficient Low Rank Adaptation of Large Models](https://mlsanity.com/paper/2402.12354/) (2024)
- [PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models](https://mlsanity.com/paper/2404.02948/) (2024)
- [LoRA-GA: Low-Rank Adaptation with Gradient Approximation](https://mlsanity.com/paper/2407.05000/) (2024)
- [No Spurious Local Minima in Nonconvex Low Rank Problems: A Unified Geometric Analysis](https://mlsanity.com/paper/1704.00708/) (2017)
- [LoRA vs Full Fine-tuning: An Illusion of Equivalence](https://mlsanity.com/paper/2410.21228/) (2024)
- [Low-Rank Few-Shot Adaptation of Vision-Language Models](https://mlsanity.com/paper/2405.18541/) (2024)
- [The Expressive Power of Low-Rank Adaptation](https://mlsanity.com/paper/2310.17513/) (2023)
- [Low-rank Solutions of Linear Matrix Equations via Procrustes Flow](https://mlsanity.com/paper/1507.03566/) (2015)
- [Asymmetry in Low-Rank Adapters of Foundation Models](https://mlsanity.com/paper/2402.16842/) (2024)
- [Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstruction](https://mlsanity.com/paper/2106.15013/) (2021)
- [LoRA-Pro: Are Low-Rank Adapters Properly Optimized?](https://mlsanity.com/paper/2407.18242/) (2024)
- [LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters](https://mlsanity.com/paper/2405.17604/) (2024)
- [SVFT: Parameter-Efficient Fine-Tuning with Singular Vectors](https://mlsanity.com/paper/2405.19597/) (2024)
- [Riemannian Preconditioned LoRA for Fine-Tuning Foundation Models](https://mlsanity.com/paper/2402.02347/) (2024)
- [LoRA Training in the NTK Regime has No Spurious Local Minima](https://mlsanity.com/paper/2402.11867/) (2024)
- [Hypernetwork-based Meta-Learning for Low-Rank Physics-Informed Neural Networks](https://mlsanity.com/paper/2310.09528/) (2023)
- [Non-square matrix sensing without spurious local minima via the Burer-Monteiro approach](https://mlsanity.com/paper/1609.03240/) (2016)
- [LoRA-One: One-Step Full Gradient Could Suffice for Fine-Tuning Large Language Models, Provably and Efficiently](https://mlsanity.com/paper/2502.01235/) (2025)
- [AdapterGNN: Parameter-Efficient Fine-Tuning Improves Generalization in GNNs](https://mlsanity.com/paper/2304.09595/) (2023)
- [Finding Low-Rank Solutions via Non-Convex Matrix Factorization, Efficiently and Provably](https://mlsanity.com/paper/1606.03168/) (2016)
- [GraphLoRA: Structure-Aware Contrastive Low-Rank Adaptation for Cross-Graph Transfer Learning](https://mlsanity.com/paper/2409.16670/) (2024)
- [Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape](https://mlsanity.com/paper/2409.14396/) (2024)
- [LoRA Training Provably Converges to a Low-Rank Global Minimum or It Fails Loudly (But it Probably Won't Fail)](https://mlsanity.com/paper/2502.09376/) (2025)
- [How Over-Parameterization Slows Down Gradient Descent in Matrix Sensing: The Curses of Symmetry and Initialization](https://mlsanity.com/paper/2310.01769/) (2023)
- [Transfer Learning in Physics-Informed Neural Networks: Full Fine-Tuning, Lightweight Fine-Tuning, and Low-Rank Adaptation](https://mlsanity.com/paper/2502.00782/) (2025)

## Abstract (arXiv)

Low-rank adaptation (LoRA) has emerged as a widely adopted parameter-efficient fine-tuning method in deep transfer learning, due to its reduced number of trainable parameters and lower memory requirements enabled by Burer-Monteiro factorization on adaptation matrices. However, classical LoRA training methods treat the low-rank factor matrices individually and optimize them using standard gradient-based algorithms. Such decoupled optimization schemes are theoretically and empirically suboptimal, as they fail to fully exploit the intrinsic structure of the LoRA parameterization. In this work, we propose a novel continuous-time optimization dynamic for LoRA factor matrices in the form of an ordinary differential equation (ODE) that emulates the gradient flow of full fine-tuning on the balanced manifold. We term this approach ODELoRA. To faithfully track the trajectories of ODELoRA, we adopt well-established and theoretically grounded time-discretization schemes, including Euler and Runge--Kutta methods. Our framework provides a unified ODE-based perspective for understanding and designing LoRA training algorithms. We establish linear convergence of the proposed method under strongly convex objectives for certain discretization schemes under mild conditions, and further extend our analysis to the matrix sensing setting. Moreover, we show that ODELoRA achieves stable feature learning, a property that is crucial for training deep neural networks at different scales of problem dimensionality. Empirical results on matrix sensing tasks confirm the derived linear convergence behavior, and experiments on training physics-informed neural networks further demonstrate the superiority of ODELoRA over existing baselines, especially in the training stability.