mlsanityData to 14 Oct 2026

LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization

arXiv 2410.20625 ↗First posted 27 Oct 2024cs.LG cs.AI cs.CLcited by 32

Not on the map. Why?

Abstract (arXiv)

Low-rank adaption (LoRA) is a widely used parameter-efficient finetuning method for LLM that reduces memory requirements. However, current LoRA optimizers lack transformation invariance, meaning the actual updates to the weights depends on how the two LoRA factors are scaled or rotated. This deficiency leads to inefficient learning and sub-optimal solutions in practice. This paper introduces LoRA-RITE, a novel adaptive matrix preconditioning method for LoRA optimization, which can achieve transformation invariance and remain computationally efficient. We provide theoretical analysis to demonstrate the benefit of our method and conduct experiments on various LLM tasks with different models including Gemma 2B, 7B, and mT5-XXL. The results demonstrate consistent improvements against existing optimizers. For example, replacing Adam with LoRA-RITE during LoRA fine-tuning of Gemma-2B yielded 4.6% accuracy gain on Super-Natural Instructions and 3.5% accuracy gain across other four LLM benchmarks (HellaSwag, ArcChallenge, GSM8K, OpenBookQA).

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Builds on 27 of its 33 references that are in the map. First posted on arXiv 27 Oct 2024.

Builds on

27 of 33 references.

References · 27

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  3. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  7. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  8. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  9. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  10. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  11. 2019Introduction to Online Convex Optimizationcited by 803
  12. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
15 more
  1. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  2. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  3. 2017Large Batch Training of Convolutional Networkscited by 535
  4. 2023Symbolic Discovery of Optimization Algorithmscited by 434
  5. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 426
  6. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 377
  7. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 233
  8. 2022SparseAdapter: An Easy Approach for Improving the Parameter-Efficiency of Adapterscited by 83
  9. 2023A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks At-Scalecited by 77
  10. 2022Revisiting Parameter-Efficient Tuning: Are We Really There Yet?cited by 76
  11. 2024A New Perspective on Shampoo's Preconditionercited by 59
  12. 2024Riemannian Preconditioned LoRA for Fine-Tuning Foundation Modelscited by 52
  13. 2023IncreLoRA: Incremental Parameter Allocation Method for Parameter-Efficient Fine-tuningcited by 45
  14. 2024Compressible Dynamics in Deep Overparameterized Low-Rank Learning & Adaptationcited by 26
  15. 2023Sketchy: Memory-efficient Adaptive Regularization with Frequent Directionscited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 14; 2026: 18.

32 papers.

Sort:

By area:

  1. 2025PoLAR: Polar-Decomposed Low-Rank Adapter Representationcited by 20
  2. 2025SubTrack++ : Gradient Subspace Tracking for Scalable LLM Trainingcited by 16
  3. 2026Muon is Not That Special: Random or Inverted Spectra Work Just as Wellcited by 14
  4. 2025LoRA meets Riemannion: Muon Optimizer for Parametrization-independent Low-Rank Adapterscited by 13
  5. 2025Preconditioned Norms: A Unified Framework for Steepest Descent, Quasi-Newton and Adaptive Methodscited by 11
  6. 2025An Overview of Low-Rank Structures in the Training and Adaptation of Large Modelscited by 9
  7. 2025AltLoRA: Towards Better Gradient Approximation in Low-Rank Adaptation with Alternating Projectionscited by 9
  8. 2025RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Modelscited by 9
  9. 2026Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuningcited by 9
  10. 2026Can Muon Fine-tune Adam-Pretrained Models?cited by 7
  11. 2025Hide & Seek: Transformer Symmetries Obscure Sharpness & Riemannian Geometry Finds Itcited by 5
  12. 2025Calibrating and Rotating: A Unified Framework for Weight Conditioning in PEFTcited by 4
20 more
  1. 2025Bi-LoRA: Efficient Sharpness-Aware Minimization for Fine-Tuning Large-Scale Modelscited by 3
  2. 2026Low-Rank Adaptation Redux for Large Modelscited by 3
  3. 2026Balanced LoRA: Removing Parameter Invariance to Accelerate Convergencecited by 3
  4. 2025SingLoRA: Low Rank Adaptation Using a Single Matrixcited by 2
  5. 2026Taming Momentum: Rethinking Optimizer States Through Low-Rank Approximationcited by 2
  6. 2026Dead-Direction Conditioners: Gauge-Equivariant Preconditioning for Deep Networkscited by 2
  7. 2026PoLoRA: A Preconditioned Orthogonalized LoRA Optimizercited by 2
  8. 2025Tuning without Peeking: Provable Generalization Bounds and Robust LLM Post-Trainingcited by 1
  9. 2025Train Large, Deploy Compact: Structured Compression for Compact Low-Rank Adaptationcited by 1
  10. 2025ScaLoRA: Optimally Scaled Low-Rank Adaptation for Efficient High-Rank Fine-Tuningcited by 1
  11. 2026SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptationcited by 1
  12. 2026Approximate Muon with low-rank adapterscited by 1
  13. 2026Stable-LoRA: Stabilizing Feature Learning of Low-Rank Adaptationcited by 0
  14. 2026AdaPreLoRA: Adafactor Preconditioned Low-Rank Adaptationcited by 0
  15. 2026PRISM: Gauge-Invariant Tangent-Space Differentially Private LoRAcited by 0
  16. 2026The Loss Does Not See the Basis, but Adam Doescited by 0
  17. 2026LoRA-TSD: Tangent-Space Spectral Descent for LoRA via Muon-Style Updatescited by 0
  18. 2026Behavior Quotient Learning for Low-Rank Adaptation of LLM Agentscited by 0
  19. 2026Stiefel-AdamW: Geometry-Aware AdamW for Linear Factorization Blockscited by 0
  20. 2026Rotated Manifold Optimization for Low-Rank Adaptationcited by 0