mlsanityData to 14 Oct 2026

Memory-Efficient LLM Training with Online Subspace Descent

arXiv 2408.12857 ↗First posted 23 Aug 2024cs.LG cs.AI cs.CLcited by 39

Not on the map. Why?

Abstract (arXiv)

Recently, a wide range of memory-efficient LLM training algorithms have gained substantial popularity. These methods leverage the low-rank structure of gradients to project optimizer states into a subspace using projection matrix found by singular value decomposition (SVD). However, convergence of these algorithms is highly dependent on the update rules of their projection matrix. In this work, we provide the first convergence guarantee for arbitrary update rules of projection matrix. This guarantee is generally applicable to optimizers that can be analyzed with Hamiltonian Descent, including most common ones, such as LION, Adam. Inspired by our theoretical understanding, we propose Online Subspace Descent, a new family of subspace descent optimizer without SVD. Instead of updating the projection matrix with eigenvectors, Online Subspace Descent updates the projection matrix with online PCA. Online Subspace Descent is flexible and introduces only minimum overhead to training. We show that for the task of pretraining LLaMA models ranging from 60M to 7B parameters on the C4 dataset, Online Subspace Descent achieves lower perplexity and better downstream tasks performance than state-of-the-art low-rank training methods across different settings and narrows the gap with full-rank baselines.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). Builds on 19 of its 24 references that are in the map. First posted on arXiv 23 Aug 2024.

Builds on

19 of 24 references.

References · 19

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  5. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  6. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  7. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  8. 2023Symbolic Discovery of Optimization Algorithmscited by 434
  9. 20218-bit Optimizers via Block-wise Quantizationcited by 403
  10. 2017TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learningcited by 256
  11. 2018Gradient Descent Happens in a Tiny Subspacecited by 217
  12. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 204
7 more
  1. 2023Full Parameter Fine-tuning for Large Language Models with Limited Resourcescited by 134
  2. 2023Lion Secretly Solves Constrained Optimization: As Lyapunov Predictscited by 51
  3. 2023MultiLoRA: Democratizing LoRA for Better Multi-Task Learningcited by 32
  4. 2021How many degrees of freedom do we need to train deep networks: a loss landscape perspectivecited by 28
  5. 2024Communication Efficient Distributed Training with Distributed Lioncited by 18
  6. 2023Sketchy: Memory-efficient Adaptive Regularization with Frequent Directionscited by 17
  7. 2019Stochastic Subspace Descentcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 18; 2026: 17.

39 papers.

Sort:

By area:

  1. 2025Muon Optimizes Under Spectral Norm Constraintscited by 86
  2. 2024APOLLO: SGD-like Memory, AdamW-level Performancecited by 54
  3. 2024Cautious Optimizers: Improving Training with One Line of Codecited by 47
  4. 2025Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noisecited by 41
  5. 2025COSMOS: A Hybrid Adaptive Optimizer for Memory-Efficient Training of LLMscited by 30
  6. 2025PoLAR: Polar-Decomposed Low-Rank Adapter Representationcited by 20
  7. 2025SubTrack++ : Gradient Subspace Tracking for Scalable LLM Trainingcited by 16
  8. 2026Training-Free Looped Transformerscited by 15
  9. 2025GaLore 2: Large-Scale LLM Pre-Training by Gradient Low-Rank Projectioncited by 14
  10. 2025Cautious Weight Decaycited by 13
  11. 2026Powering Up Zeroth-Order Training via Subspace Gradient Orthogonalizationcited by 12
  12. 2024DeMo: Decoupled Momentum Optimizationcited by 11
27 more
  1. 2025Greedy Low-Rank Gradient Compression for Distributed Learning with Convergence Guaranteescited by 11
  2. 2025Reparameterized LLM Training via Orthogonal Equivalence Transformationcited by 9
  3. 2025Breaking the Frozen Subspace: Importance Sampling for Low-Rank Optimization in LLM Pretrainingcited by 7
  4. 2025CR-Net: Scaling Parameter-Efficient Training with Cross-Layer Low-Rank Structurecited by 7
  5. 2026Do Images Speak Louder than Words? Investigating the Effect of Textual Misinformation in VLMscited by 7
  6. 2025Scalable Parameter and Memory Efficient Pretraining for LLM: Recent Algorithmic Advances and Benchmarkingcited by 5
  7. 2026POET-X: Memory-efficient LLM Training by Scaling Orthogonal Transformationcited by 5
  8. 2026φ-Balancing for Mixture-of-Experts Trainingcited by 5
  9. 2025An Efficient Subspace Algorithm for Federated Learning on Heterogeneous Datacited by 4
  10. 2026LoRIF: Low-Rank Influence Functions for Scalable Training Data Attributioncited by 3
  11. 2026Momentum Guidance: Plug-and-Play Guidance for Flow Modelscited by 3
  12. 2024Lean and Mean Adaptive Optimization via Subset-Norm and Subspace-Momentum with Convergence Guaranteescited by 2
  13. 2025FFT-based Dynamic Subspace Selection for Low-Rank Adaptive Optimization of Large Language Modelscited by 2
  14. 2025Memory-Efficient Fine-Tuning via Low-Rank Activation Compressioncited by 2
  15. 2025Memory-Efficient LLM Training by Various-Grained Low-Rank Projection of Gradientscited by 1
  16. 2025Geometrically Principled Randomized Optimization for Efficient LLM Trainingcited by 1
  17. 2026PRAC: Principal-Random Subspace for LLM Activation Compression and Memory-Efficient Trainingcited by 1
  18. 2026OASIS: Online Activation Subspace Learning for Memory-Efficient Trainingcited by 1
  19. 2026Pro-KLShampoo: Projected KL-Shampoo with Whitening Recovered by Orthogonalizationcited by 1
  20. 2025PLUMAGE: Probabilistic Low rank Unbiased Min Variance Gradient Estimator for Efficient Large Model Trainingcited by 0
  21. 2026Lotus: Efficient LLM Training by Randomized Low-Rank Gradient Projection with Adaptive Subspace Switchingcited by 0
  22. 2026From O(mn) to O(r2): Two-Sided Low-Rank Communication for Adam in Distributed Training with Memory Efficiencycited by 0
  23. 2026Optimizing the Preconditioner: A Black-box Online-to-Nonconvex Conversion with Static Regret Minimization Oraclescited by 0
  24. 2026MoARa: Module-Aware Rank Allocation and Structure-Preserving Decomposition for Low-Rank LLM Pre-trainingcited by 0
  25. 2026AIM-ZO: Activation-Informed Subspace Maintenance for Zeroth-Order LLM Fine-Tuningcited by 0
  26. 2026Clean: Second-order LLM Training at Linear Memory Cost via Nystr"om Sketchingcited by 0
  27. 2026Memory-Efficient Subspace Optimization with Residual Correction: From Momentum SGD to Muoncited by 0