mlsanityData to 14 Oct 2026

Riemannian Optimization for LoRA on the Stiefel Manifold

arXiv 2508.17901 ↗First posted 25 Aug 2025cs.LG cs.AIcited by 10

Not on the map. Why?

Abstract (arXiv)

While powerful, large language models (LLMs) present significant fine-tuning challenges due to their size. Parameter-efficient fine-tuning (PEFT) methods like LoRA provide solutions, yet suffer from critical optimizer inefficiencies; notably basis redundancy in LoRA's B matrix when using AdamW, which fundamentally limits performance. We address this by optimizing the B matrix on the Stiefel manifold, imposing explicit orthogonality constraints that achieve near-perfect orthogonality and full effective rank. This geometric approach dramatically enhances parameter efficiency and representational capacity. Our Stiefel optimizer consistently outperforms AdamW across benchmarks with both LoRA and DoRA, demonstrating that geometric constraints are the key to unlocking LoRA's full potential for effective LLM fine-tuning.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 25 of its 40 references that are in the map. First posted on arXiv 25 Aug 2025.

Builds on

25 of 40 references.

References · 25

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  10. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  11. 2014New insights and perspectives on the natural gradient methodcited by 443
  12. 2018QuAC : Question Answering in Contextcited by 381
13 more
  1. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 320
  2. 2023VeRA: Vector-based Random Matrix Adaptationcited by 300
  3. 2015Unitary Evolution Recurrent Neural Networkscited by 294
  4. 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 243
  5. 2016A Riemannian Network for SPD Matrix Learningcited by 162
  6. 2019Orthogonal Convolutional Neural Networkscited by 123
  7. 2024LoRA vs Full Fine-tuning: An Illusion of Equivalencecited by 109
  8. 2016First-order Methods for Geodesically Convex Optimizationcited by 93
  9. 2017Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networkscited by 92
  10. 2024Asymmetry in Low-Rank Adapters of Foundation Modelscited by 86
  11. 2017On orthogonality and learning recurrent networks with long term dependenciescited by 83
  12. 2023A Survey of Geometric Optimization for Deep Learning: From Euclidean Space to Riemannian Manifoldcited by 21
  13. 2025LoRAX: LoRA eXpandable Networks for Continual Synthetic Image Attributioncited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 9.

10 papers.

  1. 2026Mano: Restriking Manifold Optimization for LLM Trainingcited by 12
  2. 2026Intrinsic Muon: Spectral Optimization on Riemannian Matrix Manifoldscited by 6
  3. 2026Balanced LoRA: Removing Parameter Invariance to Accelerate Convergencecited by 3
  4. 2025Timber: Training-free Instruct Model Refining with Base via Effective Rankcited by 2
  5. 2026FoRA: Fisher-orthogonal Rank Adaptation for Parameter-Efficient Fine-Tuningcited by 1
  6. 2026Calibrated Adaptation: Bayesian Stiefel Manifold Priors for Reliable Parameter-Efficient Fine-Tuningcited by 0
  7. 2026Riemannian Stochastic Optimization for Sufficient Dimension Reductioncited by 0
  8. 2026When Top-1 Fails: Calibrating LoRA Monitors for Masked Diffusion LMscited by 0
  9. 2026Retraction-Free Optimization over the Stiefel Manifold for the LoRA Fine-Tuningcited by 0
  10. 2026Geometry-Aware Bayesian Parameter-Efficient Fine-Tuning on the Stiefel Manifold via Stein Variational Gradient Descentcited by 0