mlsanityData to 7 Oct 2026

Lotus: Efficient LLM Training by Randomized Low-Rank Gradient Projection with Adaptive Subspace Switching

arXiv 2602.01233 ↗First posted 1 Feb 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Training efficiency in large-scale models is typically assessed through memory consumption, training time, and model performance. Current methods often exhibit trade-offs among these metrics, as optimizing one generally degrades at least one of the others. Addressing this trade-off remains a central challenge in algorithm design. While GaLore enables memory-efficient training by updating gradients in a low-rank subspace, it incurs a comparable extra training time cost due to the Singular Value Decomposition(SVD) process on gradients. In this paper, we propose Lotus, a method that resolves this trade-off by simply modifying the projection process. We propose a criterion that quantifies the displacement of the unit gradient to enable efficient transitions between low-rank gradient subspaces. Experimental results indicate that Lotus is the most efficient method, achieving a 30% reduction in training time and a 40% decrease in memory consumption for gradient and optimizer states. Additionally, it outperforms the baseline method in both pre-training and fine-tuning tasks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 26 of its 28 references that are in the map. First posted on arXiv 1 Feb 2026.

Builds on

26 of 28 references.

References · 26

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  3. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  4. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  5. 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 337
  6. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  7. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 261
  8. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 200
  9. 2024Flora: Low-Rank Adapters Are Secretly Gradient Compressorscited by 103
  10. 2024SVFT: Parameter-Efficient Fine-Tuning with Singular Vectorscited by 57
  11. 2024APOLLO: SGD-like Memory, AdamW-level Performancecited by 50
  12. 2022Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equationscited by 42
14 more
  1. 2024Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint?cited by 39
  2. 2024Memory-Efficient LLM Training with Online Subspace Descentcited by 37
  3. 2025RandLoRA: Full-rank parameter-efficient fine-tuning of large modelscited by 34
  4. 2025COSMOS: A Hybrid Adaptive Optimizer for Memory-Efficient Training of LLMscited by 28
  5. 2024AdaRankGrad: Adaptive Gradient-Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuningcited by 16
  6. 2025GaLore 2: Large-Scale LLM Pre-Training by Gradient Low-Rank Projectioncited by 13
  7. 2025GoRA: Gradient-driven Adaptive Low Rank Adaptationcited by 11
  8. 2024SVFit: Parameter-Efficient Fine-Tuning of Large Pre-Trained Models Using Singular Valuescited by 10
  9. 2024Adapprox: Adaptive Approximation in Adam Optimization via Randomized Low-Rank Matricescited by 7
  10. 2025Towards Efficient Optimizer Design for LLM via Structured Fisher Approximation with a Low-Rank Extensioncited by 6
  11. 2024PMSS: Pretrained Matrices Skeleton Selection for LLM Fine-tuningcited by 5
  12. 2025CE-LoRA: Computation-Efficient LoRA Fine-Tuning for Language Modelscited by 5
  13. 2025ElaLoRA: Elastic & Learnable Low-Rank Adaptation for Efficient Model Fine-Tuningcited by 5
  14. 2025On the Duality between Gradient Transformations and Adapterscited by 1

Built on by

Not cited yet.