mlsanityData to 7 Oct 2026

LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics

arXiv 2410.16103 ↗First posted 21 Oct 2024cs.LG math.OC stat.MLcited by 35

Not on the map. Why?

Abstract (arXiv)

We introduce LDAdam, a memory-efficient optimizer for training large models, that performs adaptive optimization steps within lower dimensional subspaces, while consistently exploring the full parameter space during training. This strategy keeps the optimizer's memory footprint to a fraction of the model size. LDAdam relies on a new projection-aware update rule for the optimizer states that allows for transitioning between subspaces, i.e., estimation of the statistics of the projected gradients. To mitigate the errors due to low-rank projection, LDAdam integrates a new generalized error feedback mechanism, which explicitly accounts for both gradient and optimizer state compression. We prove the convergence of LDAdam under standard assumptions, and show that LDAdam allows for accurate and efficient fine-tuning and pre-training of language models. Code is available at https://github.com/IST-DASLab/LDAdam

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Builds on 46 of its 61 references that are in the map. First posted on arXiv 21 Oct 2024.

Builds on

46 of 61 references.

References · 46

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  10. 2020GLU Variants Improve Transformercited by 1,724
  11. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  12. 2019Root Mean Square Layer Normalizationcited by 1,425
34 more
  1. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  2. 2019On the Convergence of Adam and Beyondcited by 834
  3. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  4. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  5. 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 471
  6. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  7. 20218-bit Optimizers via Block-wise Quantizationcited by 395
  8. 2018Sparsified SGD with Memorycited by 373
  9. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  10. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  11. 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 337
  12. 2019Error Feedback Fixes SignSGD and other Gradient Compression Schemescited by 306
  13. 2019PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimizationcited by 234
  14. 2018The Convergence of Sparsified Gradient Methodscited by 228
  15. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 200
  16. 2018On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimizationcited by 173
  17. 2022Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Modelscited by 172
  18. 2020A Simple Convergence Proof of Adam and Adagradcited by 169
  19. 2023Full Parameter Fine-tuning for Large Language Models with Limited Resourcescited by 134
  20. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 134
  21. 2022Adam Can Converge Without Any Modification On Update Rulescited by 112
  22. 2023Convergence of Adam Under Relaxed Assumptionscited by 97
  23. 2018On the Convergence of Adaptive Gradient Methods for Nonconvex Optimizationcited by 91
  24. 2017Implicit Regularization in Deep Learningcited by 90
  25. 2024Chain of LoRA: Efficient Fine-tuning of Language Models via Residual Learningcited by 80
  26. 2013Accelerated Gradient Methods for Nonconvex Nonlinear and Stochastic Programmingcited by 74
  27. 2024RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptationcited by 48
  28. 2024Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradientscited by 36
  29. 2023AdaLomo: Low-memory Optimization with Adaptive Learning Ratecited by 30
  30. 2024From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applicationscited by 24
  31. 2024On Convergence of Adam for Stochastic Optimization under Relaxed Assumptionscited by 21
  32. 2022On Distributed Adaptive Optimization with Gradient Compressioncited by 19
  33. 2024MicroAdam: Accurate Adaptive Optimization with Low Space Overhead and Provable Convergencecited by 19
  34. 2023Convergence of Adam for Non-convex Objectives: Relaxed Hyperparameters and Non-ergodic Casecited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 19; 2026: 14.

35 papers.

Sort:

  1. 2024APOLLO: SGD-like Memory, AdamW-level Performancecited by 50
  2. 2025Purifying Shampoo: Investigating Shampoo's Heuristics by Decomposing its Preconditionercited by 21
  3. 2025PoLAR: Polar-Decomposed Low-Rank Adapter Representationcited by 19
  4. 2025SubTrack++ : Gradient Subspace Tracking for Scalable LLM Trainingcited by 14
  5. 2025GaLore 2: Large-Scale LLM Pre-Training by Gradient Low-Rank Projectioncited by 13
  6. 2025Greedy Low-Rank Gradient Compression for Distributed Learning with Convergence Guaranteescited by 9
  7. 2025On the Convergence Rate of LoRA Gradient Descentcited by 9
  8. 2024FRUGAL: Memory-Efficient Optimization by Reducing State Overhead for Scalable Trainingcited by 7
  9. 2025CR-Net: Scaling Parameter-Efficient Training with Cross-Layer Low-Rank Structurecited by 7
  10. 2025Scalable Parameter and Memory Efficient Pretraining for LLM: Recent Algorithmic Advances and Benchmarkingcited by 4
  11. 2025Low-rank Momentum Factorization for Memory Efficient Trainingcited by 4
  12. 2025LORENZA: Enhancing Generalization in Low-Rank Gradient LLM Training via Efficient Zeroth-Order Adaptive SAMcited by 3
23 more
  1. 2025Unified Scaling Laws for Compressed Representationscited by 3
  2. 2025Pay Attention to Small Weightscited by 3
  3. 2025FFT-based Dynamic Subspace Selection for Low-Rank Adaptive Optimization of Large Language Modelscited by 2
  4. 2025MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updatescited by 2
  5. 2026Mitigating Staleness in Asynchronous Pipeline Parallelism via Basis Rotationcited by 2
  6. 2026MGUP: A Momentum-Gradient Alignment Update Policy for Stochastic Optimizationcited by 2
  7. 2025MLorc: Momentum Low-rank Compression for Memory Efficient Large Language Model Adaptationcited by 1
  8. 2025Geometrically Principled Randomized Optimization for Efficient LLM Trainingcited by 1
  9. 2025FOAM: Blocked State Folding for Memory-Efficient LLM Trainingcited by 1
  10. 2026LoRDO: Distributed Low-Rank Optimization with Infrequent Communicationcited by 1
  11. 2026Train Less, Infer Faster: Efficient Model Finetuning and Compression via Structured Sparsitycited by 1
  12. 2026OASIS: Online Activation Subspace Learning for Memory-Efficient Trainingcited by 1
  13. 2026Subspace Optimization for Efficient Federated Learning under Heterogeneous Datacited by 1
  14. 2025GWT: Scalable Optimizer State Compression for Large Language Model Trainingcited by 0
  15. 2025Bridging Training and Merging Through Momentum-Aware Optimizationcited by 0
  16. 2026Emergent Low-Rank Training Dynamics in MLPs with Smooth Activationscited by 0
  17. 2026One Algorithm, Two Goals: Dual Scoring for Parameter and Data Selection in LLM Fine-Tuningcited by 0
  18. 2026PowerStep: Memory-Efficient Adaptive Optimization via ℓp-Norm Steepest Descentcited by 0
  19. 2026Memory-Efficient LLM Training with Dynamic Sparsity: From Stability to Practical Scalingcited by 0
  20. 2026No Subspace to Track: Non-Identifiability and Optimizer State in Low-Rank Trainingcited by 0
  21. 2026Curvature-Conditioned Multiscale Momentum with Sphere Constraints for LLM Pretrainingcited by 0
  22. 2026Blog: Survey of Optimizerscited by 0
  23. 2026MoARa: Module-Aware Rank Allocation and Structure-Preserving Decomposition for Low-Rank LLM Pre-trainingcited by 0