mlsanityData to 14 Oct 2026

AdaRankGrad: Adaptive Gradient-Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuning

arXiv 2410.17881 ↗First posted 23 Oct 2024cs.LGcited by 17

Not on the map. Why?

Abstract (arXiv)

Training and fine-tuning large language models (LLMs) come with challenges related to memory and computational requirements due to the increasing size of the model weights and the optimizer states. Various techniques have been developed to tackle these challenges, such as low-rank adaptation (LoRA), which involves introducing a parallel trainable low-rank matrix to the fixed pre-trained weights at each layer. However, these methods often fall short compared to the full-rank weight training approach, as they restrict the parameter search to a low-rank subspace. This limitation can disrupt training dynamics and require a full-rank warm start to mitigate the impact. In this paper, we introduce a new method inspired by a phenomenon we formally prove: as training progresses, the rank of the estimated layer gradients gradually decreases, and asymptotically approaches rank one. Leveraging this, our approach involves adaptively reducing the rank of the gradients during Adam optimization steps, using an efficient online-updating low-rank projections rule. We further present a randomized SVD scheme for efficiently finding the projection matrix. Our technique enables full-parameter fine-tuning with adaptive low-rank gradient updates, significantly reducing overall memory requirements during training compared to state-of-the-art methods while improving model performance in both pretraining and fine-tuning. Finally, we provide a convergence analysis of our method and demonstrate its merits for training and fine-tuning language and biological foundation models.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 32 of its 57 references that are in the map. First posted on arXiv 23 Oct 2024.

Builds on

32 of 57 references.

References · 32

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  5. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  6. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  7. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  8. 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 357
  9. 2016Deep Learning without Poor Local Minimacited by 234
  10. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 226
  11. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 204
  12. 2023Full Parameter Fine-tuning for Large Language Models with Limited Resourcescited by 134
20 more
  1. 2023A Spectral Condition for Feature Learningcited by 109
  2. 2023Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantizationcited by 100
  3. 2022SparseAdapter: An Easy Approach for Improving the Parameter-Efficiency of Adapterscited by 83
  4. 2017Training Quantized Nets: A Deeper Understandingcited by 75
  5. 2020Understanding Self-supervised Learning with Dual Deep Networkscited by 62
  6. 2024Understanding LLMs: A Comprehensive Overview from Training to Inferencecited by 59
  7. 2018SuperNeurons: Dynamic GPU Memory Management for Training Deep Neural Networkscited by 55
  8. 2023Memory Efficient Optimizers with 4-bit Statescited by 54
  9. 2024Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradientscited by 38
  10. 2023Revisiting the Noise Model of Stochastic Gradient Descentcited by 36
  11. 2023AdaLomo: Low-memory Optimization with Adaptive Learning Ratecited by 31
  12. 2021ZerO Initialization: Initializing Neural Networks with only Zeros and Onescited by 28
  13. 2020Quantized Adam with Error Feedbackcited by 27
  14. 2020Low-rank Gradient Approximation For Memory-Efficient On-device Training of Deep Neural Networkcited by 20
  15. 2024PeriodicLoRA: Breaking the Low-Rank Bottleneck in LoRA Optimizationcited by 19
  16. 2024Knowledge Editing in Language Models via Adapted Direct Preference Optimizationcited by 17
  17. 2024SLIP: Securing LLMs IP Using Weights Decompositioncited by 14
  18. 2024Adapprox: Adaptive Approximation in Adam Optimization via Randomized Low-Rank Matricescited by 8
  19. 2023Error Feedback Can Accurately Compress Preconditionerscited by 6
  20. 2022DELTA: Dynamically Optimizing GPU Memory beyond Tensor Recomputationcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 10; 2026: 6.

17 papers.

Sort:

By area:

  1. 2025SUMO: Subspace-Aware Moment-Orthogonalization for Accelerating Memory-Efficient LLM Trainingcited by 19
  2. 2026Powering Up Zeroth-Order Training via Subspace Gradient Orthogonalizationcited by 12
  3. 2025An Overview of Low-Rank Structures in the Training and Adaptation of Large Modelscited by 9
  4. 2025ICR-RL: Deep Reinforcement Learning via In-Context Regressioncited by 8
  5. 2025LORENZA: Enhancing Generalization in Low-Rank Gradient LLM Training via Efficient Zeroth-Order Adaptive SAMcited by 5
  6. 2025Conda: Column-Normalized Adam for Training Large Language Models Fastercited by 5
  7. 2025Hybrid Autoencoders for Tabular Data: Leveraging Model-Based Augmentation in Low-Label Settingscited by 5
  8. 2025Low-rank Momentum Factorization for Memory Efficient Trainingcited by 4
  9. 2024FineGates: LLMs Finetuning with Compression using Stochastic Gatescited by 3
  10. 2026Train Less, Infer Faster: Efficient Model Finetuning and Compression via Structured Sparsitycited by 2
  11. 2025Low-rank surrogate modeling and stochastic zero-order optimization for training of neural networks with black-box layerscited by 1
  12. 2025No Prior, No Leakage: Revisiting Reconstruction Attacks in Trained Neural Networkscited by 1
5 more
  1. 2025Provable Speech Attributes Conversion via Latent Independencecited by 1
  2. 2026Lotus: Efficient LLM Training by Randomized Low-Rank Gradient Projection with Adaptive Subspace Switchingcited by 0
  3. 2026Uncovering a Winning Lottery Ticket with Continuously Relaxed Bernoulli Gatescited by 0
  4. 2026Spooftral: Can Voxtral Audio-Language Model Detect Speech Spoofing?cited by 0
  5. 2026Which Optimizer Leaks Most? Optimizer Implicit Bias as a Training-Data Privacy Controllercited by 0