mlsanityData to 14 Oct 2026

Rotated Manifold Optimization for Low-Rank Adaptation

arXiv 2609.34264 ↗First posted 28 Sep 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

We propose a novel optimizer for low-rank adaptation (LoRA) that explicitly incorporates the gauge symmetry of low-rank factorization. Our optimizer extends recent matrix optimizers for full-parameter training to the manifold of fixed-rank matrices by interpreting them as normalization under a rotated basis. We show how rotation and normalization can be integrated with the fixed-rank manifold efficiently. Our optimizer converges faster to lower held-out loss and achieves better or comparable downstream performance on both supervised finetuning and reinforcement learning tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 24 of its 30 references that are in the map. First posted on arXiv 28 Sep 2026.

Builds on

24 of 30 references.

References · 24

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  9. 2021Program Synthesis with Large Language Modelscited by 3,637
  10. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  11. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  12. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
12 more
  1. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  2. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  3. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  4. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 377
  5. 2024OpenCodeInterpreter: Integrating Code Generation with Execution and Refinementcited by 247
  6. 2024Old Optimizer, New Norm: An Anthologycited by 193
  7. 2024Riemannian Preconditioned LoRA for Fine-Tuning Foundation Modelscited by 52
  8. 2024LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimizationcited by 32
  9. 2025LoRA meets Riemannion: Muon Optimizer for Parametrization-independent Low-Rank Adapterscited by 13
  10. 2026ARO: A New Lens On Matrix Optimization For Large Modelscited by 7
  11. 2026Balanced LoRA: Removing Parameter Invariance to Accelerate Convergencecited by 3
  12. 2026PoLoRA: A Preconditioned Orthogonalized LoRA Optimizercited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifold
  2. 2026Normalized Low-Rank Adaptation
  3. 2025PoLAR: Polar-Decomposed Low-Rank Adapter Representation
  4. 2024Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape
  5. 2026Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning
  6. 2026Beyond Low-Rank Parameterization: Narrowing the Gap Between LoRA and Full Fine-Tuning via Gradient Decomposition
  7. 2026SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling
  8. 2024MoR: Mixture of Ranks for Low-Rank Adaptation Tuning