mlsanityData to 7 Oct 2026

Subspace Optimization for Large Language Models with Convergence Guarantees

arXiv 2410.11289 ↗First posted 15 Oct 2024cs.LG math.OCcited by 31

Not on the map. Why?

Abstract (arXiv)

Subspace optimization algorithms, such as GaLore (Zhao et al., 2024), have gained attention for pre-training and fine-tuning large language models (LLMs) due to their memory efficiency. However, their convergence guarantees remain unclear, particularly in stochastic settings. In this paper, we reveal that GaLore does not always converge to the optimal solution and provide an explicit counterexample to support this finding. We further explore the conditions under which GaLore achieves convergence, showing that it does so when either (i) a sufficiently large mini-batch size is used or (ii) the gradient noise is isotropic. More significantly, we introduce GoLore (Gradient random Low-rank projection), a novel variant of GaLore that provably converges in typical stochastic settings, even with standard batch sizes. Our convergence analysis extends naturally to other subspace optimization algorithms. Finally, we empirically validate our theoretical results and thoroughly test the proposed mechanisms. Codes are available at https://github.com/pkumelon/Golore.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Oct 2024.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 18; 2026: 12.

31 papers.

Sort:

  1. 2024APOLLO: SGD-like Memory, AdamW-level Performancecited by 50
  2. 2025SubTrack++ : Gradient Subspace Tracking for Scalable LLM Trainingcited by 14
  3. 2026Powering Up Zeroth-Order Training via Subspace Gradient Orthogonalizationcited by 11
  4. 2025Greedy Low-Rank Gradient Compression for Distributed Learning with Convergence Guaranteescited by 9
  5. 2025On the Convergence Rate of LoRA Gradient Descentcited by 9
  6. 2025AltLoRA: Towards Better Gradient Approximation in Low-Rank Adaptation with Alternating Projectionscited by 8
  7. 2025CR-Net: Scaling Parameter-Efficient Training with Cross-Layer Low-Rank Structurecited by 7
  8. 2025Breaking the Frozen Subspace: Importance Sampling for Low-Rank Optimization in LLM Pretrainingcited by 6
  9. 2025Unbiased Gradient Low-Rank Projectioncited by 6
  10. 2025CE-LoRA: Computation-Efficient LoRA Fine-Tuning for Language Modelscited by 5
  11. 2025Scalable Parameter and Memory Efficient Pretraining for LLM: Recent Algorithmic Advances and Benchmarkingcited by 4
  12. 2025An Efficient Subspace Algorithm for Federated Learning on Heterogeneous Datacited by 4
19 more
  1. 2025AdaPM: a Partial Momentum Algorithm for LLM Trainingcited by 3
  2. 2025Memory-Efficient Differentially Private Training with Gradient Random Projectioncited by 2
  3. 2026Rethinking LoRA for Data Heterogeneous Federated Learning: Subspace and State Alignmentcited by 2
  4. 2026FedKRSO: Communication and Memory Efficient Federated Fine-Tuning of Large Language Modelscited by 2
  5. 2025MLorc: Momentum Low-rank Compression for Memory Efficient Large Language Model Adaptationcited by 1
  6. 2025Train Large, Deploy Compact: Structured Compression for Compact Low-Rank Adaptationcited by 1
  7. 2025Geometrically Principled Randomized Optimization for Efficient LLM Trainingcited by 1
  8. 2026Convergence Analysis of Randomized Subspace Normalized SGD under Heavy-Tailed Noisecited by 1
  9. 2026Omni-Masked Gradient Descent: Memory-Efficient Optimization via Mask Traversal with Improved Convergencecited by 1
  10. 2026Optimal low-rank stochastic gradient estimation for LLM trainingcited by 1
  11. 2026FedSLoP: Memory-Efficient Federated Learning with Low-Rank Gradient Projectioncited by 1
  12. 2026Pro-KLShampoo: Projected KL-Shampoo with Whitening Recovered by Orthogonalizationcited by 1
  13. 2025GWT: Scalable Optimizer State Compression for Large Language Model Trainingcited by 0
  14. 2025From PowerSGD to PowerSGD+: Low-Rank Gradient Compression for Distributed Optimization with Convergence Guaranteescited by 0
  15. 2025An All-Reduce Compatible Top-K Compressor for Communication-Efficient Distributed Learningcited by 0
  16. 2026From O(mn) to O(r2): Two-Sided Low-Rank Communication for Adam in Distributed Training with Memory Efficiencycited by 0
  17. 2026PRAC: Principal-Random Subspace for LLM Activation Compression and Memory-Efficient Trainingcited by 0
  18. 2026BROS: Bias-Corrected Randomized Subspaces for Memory-Efficient Single-Loop Bilevel Optimizationcited by 0
  19. 2026No Subspace to Track: Non-Identifiability and Optimizer State in Low-Rank Trainingcited by 0