mlsanityData to 14 Oct 2026

DeMo: Decoupled Momentum Optimization

arXiv 2411.19870 ↗First posted 29 Nov 2024cs.LG cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

Scaling neural network training increasingly depends on synchronous data-parallelism, yet full-precision gradient all-reduce imposes a severe communication bottleneck. We propose Decoupled Momentum Optimization (DeMo), a drop-in replacement for any momentum-based optimizers that significantly reduces the communication bandwidth while maintaining convergence. DeMo (i) decouples local momentum updates, (ii) applies a fast orthonormal transform (e.g., DCT) followed by top-k sparsification, and (iii) reuses the momentum buffer as error feedback via momentum subtraction. This design reduces per-step communication by up to two orders of magnitude with minimal computational overhead. Experiments on 300M and 1B-parameter DeMo language models show DeMo transmits up to 85x less data per GPU than AdamW-DDP while achieving comparable loss and accuracy. DeMo is topology-agnostic and enables training across multi-datacenter or Ethernet-based setups. Code is available at https://github.com/bloc97/DeMo

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 33 of its 51 references that are in the map. First posted on arXiv 29 Nov 2024.

Builds on

33 of 51 references.

References · 33

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  6. 2018Federated Learning with Non-IID Datacited by 1,108
  7. 2024OLMo: Accelerating the Science of Language Modelscited by 613
  8. 2018Local SGD Converges Fast and Communicates Littlecited by 605
  9. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 571
  10. 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 478
  11. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  12. 2023Symbolic Discovery of Optimization Algorithmscited by 434
21 more
  1. 2018Sparsified SGD with Memorycited by 376
  2. 2019Error Feedback Fixes SignSGD and other Gradient Compression Schemescited by 310
  3. 2017TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learningcited by 256
  4. 2017Sparse Communication for Distributed Gradient Descentcited by 249
  5. 2019PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimizationcited by 238
  6. 2018The Convergence of Sparsified Gradient Methodscited by 229
  7. 2018Don't Use Large Mini-Batches, Use Local SGDcited by 202
  8. 2024Old Optimizer, New Norm: An Anthologycited by 193
  9. 2020A Simple Convergence Proof of Adam and Adagradcited by 171
  10. 2020Memory-Efficient Pipeline-Parallel DNN Trainingcited by 134
  11. 2023DiLoCo: Distributed Low-Communication Training of Language Modelscited by 121
  12. 2024Flora: Low-Rank Adapters Are Secretly Gradient Compressorscited by 108
  13. 2021MARINA: Faster Non-Convex Distributed Learning with Compressioncited by 78
  14. 2023Lion Secretly Solves Constrained Optimization: As Lyapunov Predictscited by 51
  15. 2024Cautious Optimizers: Improving Training with One Line of Codecited by 47
  16. 2024Memory-Efficient LLM Training with Online Subspace Descentcited by 39
  17. 2019Communication-Efficient Distributed Learning via Lazily Aggregated Quantized Gradientscited by 32
  18. 2024Efficient Training of Large Language Models on Distributed Infrastructures: A Surveycited by 27
  19. 2024Communication Efficient Distributed Training with Distributed Lioncited by 18
  20. 2024Memory-Efficient Optimization with Factorized Hamiltonian Descentcited by 11
  21. 2024Communication-Efficient Federated Low-Rank Update Algorithm and its Connection to Implicit Regularizationcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 8.

11 papers.

  1. 2025Muon Optimizes Under Spectral Norm Constraintscited by 86
  2. 2025Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunchcited by 28
  3. 2026Training-Free Looped Transformerscited by 15
  4. 2026Understanding and Exploiting Weight Update Sparsity for Communication-Efficient Distributed RLcited by 7
  5. 2026φ-Balancing for Mixture-of-Experts Trainingcited by 5
  6. 2025MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updatescited by 2
  7. 2026LoRDO: Distributed Low-Rank Optimization with Infrequent Communicationcited by 1
  8. 2026Decentralised AI Training and Inference with BlockTraincited by 0
  9. 2026SCAPE: Accurate and Efficient LLM Training with Extreme Sparse Communicationcited by 0
  10. 2026Communication-Efficient LLM Adaptation over Decentralized GPU Meshescited by 0
  11. 2026Greedy Local Learning for Language Model Pretraining: Gaps and Objective Designcited by 0