mlsanityData to 14 Oct 2026

LoRA meets Riemannion: Muon Optimizer for Parametrization-independent Low-Rank Adapters

arXiv 2507.12142 ↗First posted 16 Jul 2025cs.LG cs.CL cs.NAcited by 13

Not on the map. Why?

Abstract (arXiv)

This work presents a novel, fully Riemannian framework for Low-Rank Adaptation (LoRA) that geometrically treats low-rank adapters by optimizing them directly on the fixed-rank manifold. This formulation eliminates the parametrization ambiguity present in standard Euclidean optimizers. Our framework integrates three key components to achieve this: (1) we derive Riemannion, a new Riemannian optimizer on the fixed-rank matrix manifold that generalizes the recently proposed Muon optimizer; (2) we develop a Riemannian gradient-informed LoRA initialization, and (3) we provide an efficient implementation without prominent overhead that uses automatic differentiation to compute arising geometric operations while adhering to best practices in numerical linear algebra. Comprehensive experimental results on both LLM and diffusion model architectures demonstrate that our approach yields consistent and noticeable improvements in convergence speed and final task performance over both standard LoRA and its state-of-the-art modifications.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 28 of its 50 references that are in the map. First posted on arXiv 16 Jul 2025.

Builds on

28 of 50 references.

References · 28

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  9. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  10. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  11. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  12. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
16 more
  1. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 377
  2. 2023SVDiff: Compact Parameter Space for Diffusion Fine-Tuningcited by 338
  3. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 324
  4. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 269
  5. 2023Controlling Text-to-Image Diffusion by Orthogonal Finetuningcited by 208
  6. 2023Key-Locked Rank One Editing for Text-to-Image Personalizationcited by 194
  7. 2024LoRA-GA: Low-Rank Adaptation with Gradient Approximationcited by 141
  8. 2023Parameter-Efficient Orthogonal Finetuning via Butterfly Factorizationcited by 114
  9. 2016Full-Capacity Unitary Recurrent Neural Networkscited by 94
  10. 2024MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuningcited by 94
  11. 2024Riemannian Preconditioned LoRA for Fine-Tuning Foundation Modelscited by 52
  12. 2024CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuningcited by 44
  13. 2024LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimizationcited by 32
  14. 2024Group and Shuffle: Efficient Structured Orthogonal Parametrizationcited by 11
  15. 2021Automatic differentiation for Riemannian optimization on low-rank matrix and tensor-train manifoldscited by 6
  16. 2025COALA: Numerically Stable and Efficient Framework for Context-Aware Low-Rank Approximationcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 12.

13 papers.

Sort:

By area:

  1. 2025Low-rank Orthogonalization for Large-scale Matrix Optimization with Applications to Foundation Model Trainingcited by 18
  2. 2026Intrinsic Muon: Spectral Optimization on Riemannian Matrix Manifoldscited by 6
  3. 2026Balanced LoRA: Removing Parameter Invariance to Accelerate Convergencecited by 3
  4. 2026PoLoRA: A Preconditioned Orthogonalized LoRA Optimizercited by 2
  5. 2026Tensorion: A Tensor-Aware Generalization of the Muon Optimizercited by 1
  6. 2026Approximate Muon with low-rank adapterscited by 1
  7. 2026AdaPreLoRA: Adafactor Preconditioned Low-Rank Adaptationcited by 0
  8. 2026Retraction-Free Optimization over the Stiefel Manifold for the LoRA Fine-Tuningcited by 0
  9. 2026Muon on the Stiefel Manifold Admits an Exact Closed-Form Updatecited by 0
  10. 2026LoRA-TSD: Tangent-Space Spectral Descent for LoRA via Muon-Style Updatescited by 0
  11. 2026FedSubMuon: Communication-Efficient Federated LLM Fine-Tuning via Structured Subspace Muoncited by 0
  12. 2026Behavior Quotient Learning for Low-Rank Adaptation of LLM Agentscited by 0
1 more
  1. 2026Rotated Manifold Optimization for Low-Rank Adaptationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Accelerating LLM Pre-Training through Flat-Direction Dynamics Enhancement
  2. 2026Muown Implicitly Performs Angular Step-size Decay
  3. 2025A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Models
  4. 2026Muon Meets Mamba: Spectral Optimization for State Space Models
  5. 2026Curvature-Conditioned Multiscale Momentum with Sphere Constraints for LLM Pretraining
  6. 2026MONA: Muon Optimizer with Nesterov Acceleration for Scalable Language Model Training
  7. 2026Orthogonal Witness Control for Muon Optimization via Sigmoid Spectral Reshaping
  8. 2025StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifold