mlsanityData to 7 Oct 2026

Riemannian Preconditioned LoRA for Fine-Tuning Foundation Models

arXiv 2402.02347 ↗First posted 4 Feb 2024cs.LG cs.NA math.NAcited by 51

Not on the map. Why?

Abstract (arXiv)

Low-Rank Adaptation (LoRA) emerges as a popular parameter-efficient fine-tuning (PEFT) method, which proposes to freeze pretrained model weights and update an additive low-rank trainable matrix. In this work, we study the enhancement of LoRA training by introducing an r × r preconditioner in each gradient step where r is the LoRA rank. We theoretically verify that the proposed preconditioner stabilizes feature learning with LoRA under infinite-width NN setting. Empirically, the implementation of this new preconditioner requires a small change to existing optimizer code and creates virtually minuscule storage and runtime overhead. Our experimental results with both large language models and text-to-image diffusion models show that with this new preconditioner, the convergence and reliability of SGD and AdamW can be significantly enhanced. Moreover, the training process becomes much more robust to hyperparameter choices such as learning rate. The new preconditioner can be derived from a novel Riemannian metric in low-rank matrix field. Code can be accessed at https://github.com/pilancilab/RiemannianPreconditionedLoRA.

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). Builds on 19 of its 35 references that are in the map. First posted on arXiv 4 Feb 2024.

Builds on

19 of 35 references.

References · 19

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  3. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  4. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
  5. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 229
  6. 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 146
  7. 2020DART: Open-Domain Structured Data Record to Text Generationcited by 144
  8. 2023Parameter-Efficient Orthogonal Finetuning via Butterfly Factorizationcited by 112
  9. 2019On the Impact of the Activation Function on Deep Neural Networks Trainingcited by 101
  10. 2020Neural Networks are Convex Regularizers: Exact Polynomial-time Convex Optimization Formulations for Two-layer Networkscited by 96
  11. 2023Delta-LoRA: Fine-Tuning High-Rank Parameters with the Delta of Low-Rank Matricescited by 72
  12. 2020Accelerating Ill-Conditioned Low-Rank Matrix Estimation via Scaled Gradient Descentcited by 70
7 more
  1. 2023IncreLoRA: Incremental Parameter Allocation Method for Parameter-Efficient Fine-tuningcited by 45
  2. 2022Fast Convex Optimization for Two-Layer ReLU Networks: Equivalent Model Classes and Cone Decompositionscited by 31
  3. 2021Scaling and Scalability: Provable Nonconvex Low-Rank Tensor Estimation from Incomplete Measurementscited by 28
  4. 2020Low-Rank Matrix Recovery with Scaled Subgradient Methods: Fast and Robust Convergence Without the Condition Numbercited by 27
  5. 2022Preconditioned Gradient Descent for Overparameterized Nonconvex Burer--Monteiro Factorization with Global Optimality Certificationcited by 21
  6. 2014Riemannian preconditioningcited by 19
  7. 2023Provably Accelerating Ill-Conditioned Low-rank Estimation via Scaled Gradient Descent, Even with Overparameterizationcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 16; 2026: 23.

51 papers.

Sort:

By area:

  1. 2024LoRA-Pro: Are Low-Rank Adapters Properly Optimized?cited by 70
  2. 2024A Survey on LoRA of Large Language Modelscited by 67
  3. 2024SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretrainingcited by 44
  4. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 43
  5. 2024LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimizationcited by 31
  6. 2024Spectral Adapter: Fine-Tuning in Spectral Spacecited by 21
  7. 2025PoLAR: Polar-Decomposed Low-Rank Adapter Representationcited by 19
  8. 2024On the Crucial Role of Initialization for Matrix Factorizationcited by 15
  9. 2025LoFT: Low-Rank Adaptation That Behaves Like Full Fine-Tuningcited by 13
  10. 2025LoRA meets Riemannion: Muon Optimizer for Parametrization-independent Low-Rank Adapterscited by 13
  11. 2025StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifoldcited by 13
  12. 2025A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Modelscited by 9
38 more
  1. 2025Quantum-PEFT: Ultra parameter-efficient fine-tuningcited by 9
  2. 2025RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Modelscited by 9
  3. 2025On the Convergence Rate of LoRA Gradient Descentcited by 9
  4. 2026Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuningcited by 9
  5. 2025AltLoRA: Towards Better Gradient Approximation in Low-Rank Adaptation with Alternating Projectionscited by 8
  6. 2024AutoLoRA: AutoGuidance Meets Low-Rank Adaptation for Diffusion Modelscited by 7
  7. 2025The Primacy of Magnitude in Low-Rank Adaptationcited by 7
  8. 2024No More Adam: Learning Rate Scaling at Initialization is All You Needcited by 6
  9. 2026Intrinsic Muon: Spectral Optimization on Riemannian Matrix Manifoldscited by 6
  10. 2024Guarantees of a Preconditioned Subgradient Algorithm for Overparameterized Asymmetric Low-rank Matrix Recoverycited by 4
  11. 2025MetaTT: A Global Tensor-Train Adapter for Parameter-Efficient Fine-Tuningcited by 4
  12. 2025Calibrating and Rotating: A Unified Framework for Weight Conditioning in PEFTcited by 4
  13. 2026Rethinking LoRA for Privacy-Preserving Federated Learning in Large Modelscited by 4
  14. 2024ALLoRA: Adaptive Learning Rate Mitigates LoRA Fatal Flawscited by 2
  15. 2025Elucidating Subspace Perturbation in Zeroth-Order Optimization: Theory and Practice at Scalecited by 2
  16. 2025SingLoRA: Low Rank Adaptation Using a Single Matrixcited by 2
  17. 2026A Unified Study of LoRA Variants: Taxonomy, Review, Codebase, and Empirical Evaluationcited by 2
  18. 2026Low-Rank Adaptation Redux for Large Modelscited by 2
  19. 2026Balanced LoRA: Removing Parameter Invariance to Accelerate Convergencecited by 2
  20. 2024Collaborative and Efficient Personalization with Mixtures of Adaptorscited by 1
  21. 2025Faster Than SVD, Smarter Than SGD: The OPLoRA Alternating Updatecited by 1
  22. 2026Curvature-Guided LoRA: Matching Full Fine-Tuning in Function Spacecited by 1
  23. 2026On the Convergence of Stochastic Low-Rank Adaptationcited by 1
  24. 2024OP-LoRA: The Blessing of Dimensionalitycited by 0
  25. 2026Diving into Kronecker Adapters: Component Design Matterscited by 0
  26. 2026ODELoRA: Training Low-Rank Adaptation by Solving Ordinary Differential Equationscited by 0
  27. 2026Beyond SGD, Without SVD: Proximal Subspace Iteration LoRA with Diagonal Fractional K-FACcited by 0
  28. 2026Calibrated Adaptation: Bayesian Stiefel Manifold Priors for Reliable Parameter-Efficient Fine-Tuningcited by 0
  29. 2026Stable-LoRA: Stabilizing Feature Learning of Low-Rank Adaptationcited by 0
  30. 2026GiVA: Gradient-Informed Bases for Vector-Based Adaptationcited by 0
  31. 2026AdaPreLoRA: Adafactor Preconditioned Low-Rank Adaptationcited by 0
  32. 2026RotMoLE: Enhancing Mixture of Low-Rank Experts through Rotational Gating Mechanismcited by 0
  33. 2026Finer Parameter Steps for Low-Rank PEFT: A Controlled Study with CP Tensor Adapterscited by 0
  34. 2026SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptationcited by 0
  35. 2026Approximate Muon with low-rank adapterscited by 0
  36. 2026LoRA-TSD: Tangent-Space Spectral Descent for LoRA via Muon-Style Updatescited by 0
  37. 2026Behavior Quotient Learning for Low-Rank Adaptation of LLM Agentscited by 0
  38. 2026Stiefel-AdamW: Geometry-Aware AdamW for Linear Factorization Blockscited by 0

50 most cited of 51 All 51 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026FuRA: Full-Rank Parameter-Efficient Fine-Tuning with Spectral Preconditioning
  2. 2026Beyond Low-Rank Parameterization: Narrowing the Gap Between LoRA and Full Fine-Tuning via Gradient Decomposition
  3. 2024LoLDU: Low-Rank Adaptation via Lower-Diag-Upper Decomposition for Parameter-Efficient Fine-Tuning
  4. 2025Mitigating Forgetting in Low Rank Adaptation
  5. 2024Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape
  6. 2023The Expressive Power of Low-Rank Adaptation
  7. 2025GraLoRA: Granular Low-Rank Adaptation for Parameter-Efficient Fine-Tuning
  8. 2025Revisiting LoRA through the Lens of Parameter Redundancy: Spectral Encoding Helps