mlsanityData to 7 Oct 2026

GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection

arXiv 2403.03507 ↗First posted 6 Mar 2024cs.LGcited by 447

Not on the map. Why?

Abstract (arXiv)

Training Large Language Models (LLMs) presents significant memory challenges, predominantly due to the growing size of weights and optimizer states. Common memory-reduction approaches, such as low-rank adaptation (LoRA), add a trainable low-rank matrix to the frozen pre-trained weight in each layer, reducing trainable parameters and optimizer states. However, such approaches typically underperform training with full-rank weights in both pre-training and fine-tuning stages since they limit the parameter search to a low-rank subspace and alter the training dynamics, and further, may require full-rank warm start. In this work, we propose Gradient Low-Rank Projection (GaLore), a training strategy that allows full-parameter learning but is more memory-efficient than common low-rank adaptation methods such as LoRA. Our approach reduces memory usage by up to 65.5% in optimizer states while maintaining both efficiency and performance for pre-training on LLaMA 1B and 7B architectures with C4 dataset with up to 19.7B tokens, and on fine-tuning RoBERTa on GLUE tasks. Our 8-bit GaLore further reduces optimizer memory by up to 82.5% and total training memory by 63.3%, compared to a BF16 baseline. Notably, we demonstrate, for the first time, the feasibility of pre-training a 7B model on consumer GPUs with 24GB memory (e.g., NVIDIA RTX 4090) without model parallel, checkpointing, or offloading strategies.

Key facts. Cited by 447 papers in the map (data through 7 Oct 2026). Builds on 46 of its 57 references that are in the map. First posted on arXiv 6 Mar 2024.

Builds on

46 of 57 references.

References · 46

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  9. 2020GLU Variants Improve Transformercited by 1,724
  10. 2019Root Mean Square Layer Normalizationcited by 1,425
  11. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  12. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
34 more
  1. 2021Scaling Vision Transformerscited by 993
  2. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  3. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  4. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  5. 20218-bit Optimizers via Block-wise Quantizationcited by 395
  6. 2018Gradient Descent Happens in a Tiny Subspacecited by 208
  7. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 200
  8. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 179
  9. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 146
  10. 2018ATOMO: Communication-efficient Learning via Atomic Sparsificationcited by 135
  11. 2023Full Parameter Fine-tuning for Large Language Models with Limited Resourcescited by 134
  12. 2023LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuningcited by 132
  13. 2024Flora: Low-Rank Adapters Are Secretly Gradient Compressorscited by 103
  14. 2023A Spectral Condition for Feature Learningcited by 101
  15. 2020Continual Learning in Low-rank Orthogonal Subspacescited by 99
  16. 2024Chain of LoRA: Efficient Fine-tuning of Language Models via Residual Learningcited by 80
  17. 2023Stable and low-precision training for large-scale vision-language modelscited by 74
  18. 2015Fast low-rank estimation by projected gradient descent: General statistical and algorithmic guaranteescited by 68
  19. 2023Tied-Lora: Enhancing parameter efficiency of LoRA with weight tyingcited by 65
  20. 2020Understanding Self-supervised Learning with Dual Deep Networkscited by 62
  21. 2023Memory Efficient Optimizers with 4-bit Statescited by 54
  22. 2023JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and Attentioncited by 52
  23. 2020PowerGossip: Practical Low-Rank Communication Compression in Decentralized Deep Learningcited by 35
  24. 2023MultiLoRA: Democratizing LoRA for Better Multi-Task Learningcited by 32
  25. 2022Exploring Low Rank Training of Deep Neural Networkscited by 30
  26. 2023AdaLomo: Low-memory Optimization with Adaptive Learning Ratecited by 30
  27. 2021How many degrees of freedom do we need to train deep networks: a loss landscape perspectivecited by 28
  28. 2021ZerO Initialization: Initializing Neural Networks with only Zeros and Onescited by 28
  29. 2023Cuttlefish: Low-Rank Model Training without All the Tuningcited by 25
  30. 2016Non-Convex Projected Gradient Descent for Generalized Low-Rank Tensor Regressioncited by 23
  31. 2020Low-rank Gradient Approximation For Memory-Efficient On-device Training of Deep Neural Networkcited by 20
  32. 2023InRank: Incremental Low-Rank Learningcited by 10
  33. 2019Dynamic Mini-batch SGD for Elastic Distributed Training: Learning in the Limbo of Resourcescited by 7
  34. 2023Error Feedback Can Accurately Compress Preconditionerscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 120; 2025: 167; 2026: 160.

447 papers.

Sort:

By area:

  1. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  2. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  3. 2024LoRA Learns Less and Forgets Lesscited by 313
  4. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  5. 2024SOAP: Improving and Stabilizing Shampoo using Adamcited by 179
  6. 2024SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Modelscited by 161
  7. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 134
  8. 2024LoRA-GA: Low-Rank Adaptation with Gradient Approximationcited by 134
  9. 2024On-Device Language Models: A Comprehensive Reviewcited by 101
  10. 2024LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuningcited by 88
  11. 2025NorMuon: Making Muon more efficient and scalablecited by 74
  12. 2024LoRA-Pro: Are Low-Rank Adapters Properly Optimized?cited by 70
38 more
  1. 2024A Survey on LoRA of Large Language Modelscited by 67
  2. 2025Dion: Distributed Orthonormalized Updatescited by 64
  3. 2024Palu: Compressing KV-Cache with Low-Rank Projectioncited by 59
  4. 2025The Curse of Depth in Large Language Modelscited by 58
  5. 2025ASGO: Adaptive Structured Gradient Optimizationcited by 55
  6. 2024Lean Copilot: Large Language Models as Copilots for Theorem Proving in Leancited by 52
  7. 2025The Path Not Taken: RLVR Provably Learns Off the Principalscited by 52
  8. 2024APOLLO: SGD-like Memory, AdamW-level Performancecited by 50
  9. 2024Understanding the Performance and Estimating the Cost of LLM Fine-Tuningcited by 48
  10. 2024Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structurescited by 48
  11. 2024SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretrainingcited by 44
  12. 2024Med-MoE: Mixture of Domain-Specific Experts for Lightweight Medical Vision-Language Modelscited by 43
  13. 2024Cautious Optimizers: Improving Training with One Line of Codecited by 42
  14. 2024CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuningcited by 41
  15. 2024Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint?cited by 39
  16. 2024Memory-Efficient LLM Training with Online Subspace Descentcited by 37
  17. 2024Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LNcited by 37
  18. 2024Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradientscited by 36
  19. 2024LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statisticscited by 35
  20. 2024LeanAgent: Lifelong Learning for Formal Theorem Provingcited by 34
  21. 2024COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Trainingcited by 32
  22. 2024On the Impact of Fine-Tuning on Chain-of-Thought Reasoningcited by 32
  23. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  24. 2024Online Merging Optimizers for Boosting Rewards and Mitigating Tax in Alignmentcited by 28
  25. 2025CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activationcited by 28
  26. 2025COSMOS: A Hybrid Adaptive Optimizer for Memory-Efficient Training of LLMscited by 28
  27. 2024BAdam: A Memory Efficient Full Parameter Optimization Method for Large Language Modelscited by 27
  28. 2025SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Trainingcited by 27
  29. 2025The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Trainingcited by 27
  30. 2025Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunchcited by 26
  31. 2025WikiBigEdit: Understanding the Limits of Lifelong Knowledge Editing in LLMscited by 25
  32. 2025TorchAO: PyTorch-Native Training-to-Serving Model Optimizationcited by 25
  33. 2024Compute Better Spent: Replacing Dense Layers with Structured Matricescited by 24
  34. 2024From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applicationscited by 24
  35. 2024Zeroth-Order Fine-Tuning of LLMs in Random Subspacescited by 23
  36. 2024Reassessing Layer Pruning in LLMs: New Insights and Methodscited by 23
  37. 202570% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11)cited by 23
  38. 2024A Practitioner's Guide to Continual Multimodal Pretrainingcited by 22

50 most cited of 447 All 447 (JSON)