mlsanityData to 14 Oct 2026

SubTrack++ : Gradient Subspace Tracking for Scalable LLM Training

arXiv 2502.01586 ↗First posted 3 Feb 2025cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

Training large language models (LLMs) is highly resource-intensive due to their massive number of parameters and the overhead of optimizer states. While recent work has aimed to reduce memory consumption, such efforts often entail trade-offs among memory efficiency, training time, and model performance. Yet, true democratization of LLMs requires simultaneous progress across all three dimensions. To this end, we propose SubTrack++ that leverages Grassmannian gradient subspace tracking combined with projection-aware optimizers, enabling Adam's internal statistics to adapt to subspace changes. Additionally, employing recovery scaling, a technique that restores information lost through low-rank projections, further enhances model performance. Our method demonstrates SOTA convergence by exploiting Grassmannian geometry, reducing pre-training wall-time by up to 65% and fine-tuning time by 36% compared to existing SOTA methods, while maintaining the same memory footprint.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 35 of its 55 references that are in the map. First posted on arXiv 3 Feb 2025.

Builds on

35 of 55 references.

References · 35

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  3. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  4. 2019SuperGlue: Learning Feature Matching with Graph Neural Networkscited by 1,443
  5. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  6. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  7. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  8. 20218-bit Optimizers via Block-wise Quantizationcited by 403
  9. 2018Gradient Descent Happens in a Tiny Subspacecited by 217
  10. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 204
  11. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 137
  12. 2023Full Parameter Fine-tuning for Large Language Models with Limited Resourcescited by 134
23 more
  1. 2024Flora: Low-Rank Adapters Are Secretly Gradient Compressorscited by 108
  2. 2024LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuningcited by 89
  3. 2024Chain of LoRA: Efficient Fine-tuning of Language Models via Residual Learningcited by 82
  4. 2023Tied-Lora: Enhancing parameter efficiency of LoRA with weight tyingcited by 65
  5. 2023Memory Efficient Optimizers with 4-bit Statescited by 54
  6. 2024Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint?cited by 43
  7. 2024Memory-Efficient LLM Training with Online Subspace Descentcited by 39
  8. 2024LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statisticscited by 38
  9. 2024Subspace Optimization for Large Language Models with Convergence Guaranteescited by 34
  10. 2024LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimizationcited by 32
  11. 2024BAdam: A Memory Efficient Full Parameter Optimization Method for Large Language Modelscited by 27
  12. 2024Compressible Dynamics in Deep Overparameterized Low-Rank Learning & Adaptationcited by 26
  13. 2017Robust Subspace Learning: Robust PCA, Robust Subspace Tracking, and Robust Subspace Recoverycited by 25
  14. 2018Streaming PCA and Subspace Tracking: The Missing Data Casecited by 19
  15. 2018Grassmannian Learning: Embedding Geometry Awareness in Shallow and Deep Learningcited by 19
  16. 2024MicroAdam: Accurate Adaptive Optimization with Low Space Overhead and Provable Convergencecited by 19
  17. 2024VeLoRA: Memory Efficient Training using Rank-1 Sub-Token Projectionscited by 18
  18. 2024Grass: Compute Efficient Low-Memory LLM Training with Structured Sparse Gradientscited by 18
  19. 2024BlockLLM: Memory-Efficient Adaptation of LLMs by Selecting and Optimizing the Right Coordinate Blockscited by 9
  20. 2015Global Convergence of a Grassmannian Gradient Descent Algorithm for Subspace Estimationcited by 5
  21. 2024Identifying Policy Gradient Subspacescited by 5
  22. 2017Intrinsic Grassmann Averages for Online Linear, Robust and Nonlinear Subspace Learningcited by 4
  23. 2017Fast online low-rank tensor subspace tracking by CP decomposition using recursive least squares from incomplete observationscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 12.

16 papers.

Sort:

By area:

  1. 2026Mano: Restriking Manifold Optimization for LLM Trainingcited by 12
  2. 2025LORENZA: Enhancing Generalization in Low-Rank Gradient LLM Training via Efficient Zeroth-Order Adaptive SAMcited by 5
  3. 2025Scalable Parameter and Memory Efficient Pretraining for LLM: Recent Algorithmic Advances and Benchmarkingcited by 5
  4. 2026TamperBench: Systematically Stress-Testing LLM Safety Under Fine-Tuning and Tamperingcited by 2
  5. 2025Geometrically Principled Randomized Optimization for Efficient LLM Trainingcited by 1
  6. 2025FOAM: Blocked State Folding for Memory-Efficient LLM Trainingcited by 1
  7. 2026Do Prompts Guarantee Safety? Mitigating Toxicity from LLM Generations through Subspace Interventioncited by 1
  8. 2026OASIS: Online Activation Subspace Learning for Memory-Efficient Trainingcited by 1
  9. 2026Pro-KLShampoo: Projected KL-Shampoo with Whitening Recovered by Orthogonalizationcited by 1
  10. 2026Emergent Low-Rank Training Dynamics in MLPs with Smooth Activationscited by 0
  11. 2026No Subspace to Track: Non-Identifiability and Optimizer State in Low-Rank Trainingcited by 0
  12. 2026FedSubMuon: Communication-Efficient Federated LLM Fine-Tuning via Structured Subspace Muoncited by 0
4 more
  1. 2026MoARa: Module-Aware Rank Allocation and Structure-Preserving Decomposition for Low-Rank LLM Pre-trainingcited by 0
  2. 2026AIM-ZO: Activation-Informed Subspace Maintenance for Zeroth-Order LLM Fine-Tuningcited by 0
  3. 2026Clean: Second-order LLM Training at Linear Memory Cost via Nystr"om Sketchingcited by 0
  4. 2026Do More Modalities Always Help? A Geometric Perspective on Missing-Modality Robustnesscited by 0