mlsanityData to 14 Oct 2026

AdaPM: a Partial Momentum Algorithm for LLM Training

arXiv 2510.09103 ↗First posted 10 Oct 2025cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

In the training of large language models, momentum is widely used and often demonstrated to achieve significant acceleration. However, storing momentum typically presents memory challenges. In this paper, we propose AdaPM, an adaptive training strategy that leverages partial momentum to implement a memory-efficient optimizer. To this end, AdaPM utilizes a non-uniform momentum design: for most blocks, full momentum is not necessary to preserve the performance of the optimization. In the momentum design of AdaPM, to mitigate the bias and performance loss caused by partial momentum, we enhance the partial momentum by a bias correction technique. Empirically, we verify that our approach reduces memory by over 90% in momentum while maintaining both efficiency and performance for pretraining various language models ranging from 60M to 1.5B, as well as for supervised fine-tuning and RLHF. AdaPM can further reduce memory by up to 95% in optimizer states by combining the memory-efficient technique on the second-order statistic, saving over 30% GPU hours for pretraining GPT-2 1.5B.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 29 of its 41 references that are in the map. First posted on arXiv 10 Oct 2025.

Builds on

29 of 41 references.

References · 29

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2017Proximal Policy Optimization Algorithmscited by 15,215
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  9. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  10. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  11. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  12. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
17 more
  1. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  2. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 265
  3. 2018Gradient Descent Happens in a Tiny Subspacecited by 217
  4. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 204
  5. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 187
  6. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 137
  7. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 124
  8. 2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
  9. 2023A Spectral Condition for Feature Learningcited by 109
  10. 2020Continual Learning in Low-rank Orthogonal Subspacescited by 99
  11. 2023CAME: Confidence-guided Adaptive Memory Efficient Optimizationcited by 36
  12. 2024Subspace Optimization for Large Language Models with Convergence Guaranteescited by 34
  13. 2022Neural Tangent Kernel: A Surveycited by 23
  14. 2020Low-rank Gradient Approximation For Memory-Efficient On-device Training of Deep Neural Networkcited by 20
  15. 2023When and Why Momentum Accelerates SGD:An Empirical Studycited by 19
  16. 2023Risk Bounds of Accelerated SGD for Overparameterized Linear Regressioncited by 7
  17. 2025Optimal Algorithms in Linear Regression under Covariate Shift: On the Importance of Preconditioncited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Taming Momentum: Rethinking Optimizer States Through Low-Rank Approximationcited by 2
  2. 2026PRAC: Principal-Random Subspace for LLM Activation Compression and Memory-Efficient Trainingcited by 1
  3. 2026Memory-Efficient LLM Training with Dynamic Sparsity: From Stability to Practical Scalingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025AdamS: Momentum Itself Can Be A Normalizer for LLM Pretraining and Post-training
  2. 2026Out of the Memory Barrier: A Highly Memory Efficient Training System for LLMs with Million-Token Contexts
  3. 2026SNIP: An Adaptive Mixed Precision Framework for Subbyte Large Language Model Training
  4. 2024APOLLO: SGD-like Memory, AdamW-level Performance
  5. 2025Predictable Scale: Part I, Step Law -- Optimal Hyperparameter Scaling Law in Large Language Model Pretraining
  6. 2026Efficient Scaling of LLM Training with Flexible Context Parallelism
  7. 2023Dissecting the Runtime Performance of the Training, Fine-tuning, and Inference of Large Language Models
  8. 2026SCAPE: Accurate and Efficient LLM Training with Extreme Sparse Communication