mlsanityData to 7 Oct 2026

GaLore 2: Large-Scale LLM Pre-Training by Gradient Low-Rank Projection

arXiv 2504.20437 ↗First posted 29 Apr 2025cs.LG cs.AIcited by 13

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have revolutionized natural language understanding and generation but face significant memory bottlenecks during training. GaLore, Gradient Low-Rank Projection, addresses this issue by leveraging the inherent low-rank structure of weight gradients, enabling substantial memory savings without sacrificing performance. Recent works further extend GaLore from various aspects, including low-bit quantization and higher-order tensor structures. However, there are several remaining challenges for GaLore, such as the computational overhead of SVD for subspace updates and the integration with state-of-the-art training parallelization strategies (e.g., FSDP). In this paper, we present GaLore 2, an efficient and scalable GaLore framework that addresses these challenges and incorporates recent advancements. In addition, we demonstrate the scalability of GaLore 2 by pre-training Llama 7B from scratch using up to 500 billion training tokens, highlighting its potential impact on real LLM pre-training scenarios.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 33 of its 37 references that are in the map. First posted on arXiv 29 Apr 2025.

Builds on

33 of 37 references.

References · 33

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  6. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 763
  7. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  8. 2024SpinQuant: LLM quantization with learned rotationscited by 446
  9. 20218-bit Optimizers via Block-wise Quantizationcited by 395
  10. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 202
  11. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 200
  12. 2024AdvPrompter: Fast Adaptive Adversarial Prompting for LLMscited by 146
21 more
  1. 2025SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Taskscited by 108
  2. 2016TensorLy: Tensor Learning in Pythoncited by 104
  3. 2024Beyond A*: Better Planning with Transformers via Search Dynamics Bootstrappingcited by 89
  4. 2024LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuningcited by 88
  5. 2025Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoningcited by 87
  6. 2023Guiding Language Model Reasoning with Planning Tokenscited by 58
  7. 2024Dualformer: Controllable Fast and Slow Thinking by Learning with Randomized Reasoning Tracescited by 40
  8. 2024Memory-Efficient LLM Training with Online Subspace Descentcited by 37
  9. 2024Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradientscited by 36
  10. 2024LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statisticscited by 35
  11. 2023MultiLoRA: Democratizing LoRA for Better Multi-Task Learningcited by 32
  12. 2023AdaLomo: Low-memory Optimization with Adaptive Learning Ratecited by 30
  13. 2023Learning Personalized Alignment for Evaluating Open-ended Text Generationcited by 29
  14. 2024From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applicationscited by 24
  15. 2025Step-KTO: Optimizing Mathematical Reasoning through Stepwise Binary Feedbackcited by 22
  16. 2022Narrowing the Coordinate-frame Gap in Behavior Prediction Models: Distillation for Efficient and Accurate Scene-centric Motion Forecastingcited by 10
  17. 2025Spectral Journey: How Transformers Predict the Shortest Pathcited by 8
  18. 2025TensorGRaD: Tensor Gradient Robust Decomposition for Memory-Efficient Neural Operator Trainingcited by 5
  19. 2020ConQUR: Mitigating Delusional Bias in Deep Q-learningcited by 4
  20. 2021MUSBO: Model-based Uncertainty Regularized and Sample Efficient Batch Optimization for Deployment Constrained Reinforcement Learningcited by 4
  21. 2024Natural GaLore: Accelerating GaLore for memory-efficient LLM Training and Fine-tuningcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 10.

13 papers.

Sort:

By area:

  1. 2025Purifying Shampoo: Investigating Shampoo's Heuristics by Decomposing its Preconditionercited by 21
  2. 2026The Newton-Muon Optimizercited by 14
  3. 2025Reparameterized LLM Training via Orthogonal Equivalence Transformationcited by 8
  4. 2026Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routerscited by 5
  5. 2025Scalable Parameter and Memory Efficient Pretraining for LLM: Recent Algorithmic Advances and Benchmarkingcited by 4
  6. 2026POET-X: Memory-efficient LLM Training by Scaling Orthogonal Transformationcited by 3
  7. 2026Rethinking LoRA for Data Heterogeneous Federated Learning: Subspace and State Alignmentcited by 2
  8. 2026Gradient Clipping Beyond Vector Norms: A Spectral Approach for Matrix-Valued Parameterscited by 2
  9. 2026Lotus: Efficient LLM Training by Randomized Low-Rank Gradient Projection with Adaptive Subspace Switchingcited by 0
  10. 2026From O(mn) to O(r2): Two-Sided Low-Rank Communication for Adam in Distributed Training with Memory Efficiencycited by 0
  11. 2026Beyond Perplexity: A Geometric and Spectral Study of Low-Rank Pre-Trainingcited by 0
  12. 2026No Subspace to Track: Non-Identifiability and Optimizer State in Low-Rank Trainingcited by 0
1 more
  1. 2026Blog: Survey of Optimizerscited by 0