# LoRA-One: One-Step Full Gradient Could Suffice for Fine-Tuning Large Language Models, Provably and Efficiently

arXiv: https://arxiv.org/abs/2502.01235 · HTML: https://mlsanity.com/paper/2502.01235/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Feb 2025.

## Built on by (most cited first, 30 of 38)

- [On the Convergence Rate of LoRA Gradient Descent](https://mlsanity.com/paper/2512.18248/) (2025)
- [Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning](https://mlsanity.com/paper/2602.04998/) (2026)
- [AltLoRA: Towards Better Gradient Approximation in Low-Rank Adaptation with Alternating Projections](https://mlsanity.com/paper/2505.12455/) (2025)
- [Can Muon Fine-tune Adam-Pretrained Models?](https://mlsanity.com/paper/2605.10468/) (2026)
- [Channel Location Constrains the Auditability of Subliminal Learning](https://mlsanity.com/paper/2606.22019/) (2026)
- [Model Merging with Functional Dual Anchors](https://mlsanity.com/paper/2510.21223/) (2025)
- [LoRA-DA: Data-Aware Initialization for Low-Rank Adaptation via Asymptotic Analysis](https://mlsanity.com/paper/2510.24561/) (2025)
- [ABBA-Adapters: Efficient and Expressive Fine-Tuning of Foundation Models](https://mlsanity.com/paper/2505.14238/) (2025)
- [Low-Rank Adaptation Redux for Large Models](https://mlsanity.com/paper/2604.21905/) (2026)
- [High-Dimensional Theory of LoRA Fine-Tuning in a Solvable Attention Model](https://mlsanity.com/paper/2606.05899/) (2026)
- [DiaBlo: Diagonal Blocks Are Sufficient For Finetuning](https://mlsanity.com/paper/2506.03230/) (2025)
- [Multiple Choice Learning of Low-Rank Adapters for Language Modeling](https://mlsanity.com/paper/2507.10419/) (2025)
- [Neural Collapse under Gradient Flow on Shallow ReLU Networks for Orthogonally Separable Data](https://mlsanity.com/paper/2510.21078/) (2025)
- [A Unified Study of LoRA Variants: Taxonomy, Review, Codebase, and Empirical Evaluation](https://mlsanity.com/paper/2601.22708/) (2026)
- [When pre-training hurts LoRA fine-tuning: a dynamical analysis via single-index models](https://mlsanity.com/paper/2602.02855/) (2026)
- [LoRA vs. Full Fine-Tuning: A Theoretical Perspective](https://mlsanity.com/paper/2605.19018/) (2026)
- [Train Large, Deploy Compact: Structured Compression for Compact Low-Rank Adaptation](https://mlsanity.com/paper/2510.00192/) (2025)
- [Activation Boundary Matching: Task-Informed Initialization for Low-Rank Adaptation](https://mlsanity.com/paper/2511.19145/) (2025)
- [Curvature-Guided LoRA: Matching Full Fine-Tuning in Function Space](https://mlsanity.com/paper/2603.29824/) (2026)
- [PaperFit: Vision-in-the-Loop Typesetting Optimization for Scientific Documents](https://mlsanity.com/paper/2605.10341/) (2026)
- [SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation](https://mlsanity.com/paper/2606.16454/) (2026)
- [On the Convergence of Stochastic Low-Rank Adaptation](https://mlsanity.com/paper/2607.21975/) (2026)
- [IPA: An Information-Reconstructive Input Projection Framework for Efficient Foundation Model Adaptation](https://mlsanity.com/paper/2509.04398/) (2025)
- [Rethinking Fine-Tuning: Unlocking Hidden Capabilities in Vision-Language Models](https://mlsanity.com/paper/2512.23073/) (2025)
- [Diving into Kronecker Adapters: Component Design Matters](https://mlsanity.com/paper/2602.01267/) (2026)
- [ODELoRA: Training Low-Rank Adaptation by Solving Ordinary Differential Equations](https://mlsanity.com/paper/2602.07479/) (2026)
- [SOLAR: Communication-Efficient Model Adaptation via Subspace-Oriented Latent Adapter Reparametrization](https://mlsanity.com/paper/2604.08368/) (2026)
- [GiVA: Gradient-Informed Bases for Vector-Based Adaptation](https://mlsanity.com/paper/2604.21901/) (2026)
- [AdaPreLoRA: Adafactor Preconditioned Low-Rank Adaptation](https://mlsanity.com/paper/2605.08734/) (2026)
- [Between Gradient and Natural Gradient: A Continuum of LoRA Initializations](https://mlsanity.com/paper/2607.26247/) (2026)

## Abstract (arXiv)

This paper explores how theory can guide and enhance practical algorithms, using Low-Rank Adaptation (LoRA, Hu et al. 2022) in large language models as a case study. We rigorously prove that, under gradient descent, LoRA adapters align with specific singular subspaces of the one-step full fine-tuning gradient. This result suggests that, by properly initializing the adapters using the one-step full gradient, subspace alignment can be achieved immediately and applicable to both linear and nonlinear models. Building on our theory, we propose a theory-driven algorithm, LoRA-One, where the linear convergence (as well as generalization) is built and incorporating preconditioners theoretically helps mitigate the effects of ill-conditioning. Besides, our theory reveals connections between LoRA-One and other gradient-alignment-based methods, helping to clarify misconceptions in the design of such algorithms. LoRA-One achieves significant empirical improvements over LoRA and its variants across benchmarks in natural language understanding, mathematical reasoning, and code generation. Code is available at: https://github.com/YuanheZ/LoRA-One.