mlsanityData to 14 Oct 2026

From promise to practice: realizing high-performance decentralized training

arXiv 2410.11998 ↗First posted 15 Oct 2024cs.DC cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Decentralized training of deep neural networks has attracted significant attention for its theoretically superior scalability over synchronous data-parallel methods like All-Reduce. However, realizing this potential in multi-node training is challenging due to the complex design space that involves communication topologies, computation patterns, and optimization algorithms. This paper identifies three key factors that can lead to speedups over All-Reduce training and constructs a runtime model to determine when, how, and to what degree decentralization can yield shorter per-iteration runtimes. Furthermore, to support the decentralized training of transformer-based models, we study a decentralized Adam algorithm that allows for overlapping communications and computations, prove its convergence, and propose an accumulation technique to mitigate the high variance caused by small local batch sizes. We deploy the proposed approach in clusters with up to 64 GPUs and demonstrate its practicality and advantages in both runtime and generalization performance under a fixed iteration budget.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 36 of its 56 references that are in the map. First posted on arXiv 15 Oct 2024.

Builds on

36 of 56 references.

References · 36

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  7. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  8. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,475
  9. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  10. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  11. 2017Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descentcited by 502
  12. 2018Horovod: fast and easy distributed deep learning in TensorFlowcited by 279
24 more
  1. 2019PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimizationcited by 238
  2. 2020A Simple Convergence Proof of Adam and Adagradcited by 171
  3. 2013On the Convergence of Decentralized Gradient Descentcited by 159
  4. 2018Stochastic Gradient Push for Distributed Deep Learningcited by 152
  5. 2016Achieving Geometric Convergence for Distributed Optimization over Time-Varying Graphscited by 150
  6. 2017Network Topology and Communication-Computation Tradeoffs in Decentralized Optimizationcited by 122
  7. 2019SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentumcited by 121
  8. 2018Communication Compression for Decentralized Trainingcited by 88
  9. 2022Decentralized Training of Foundation Models in Heterogeneous Environmentscited by 87
  10. 2021Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Datacited by 69
  11. 2021Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelinescited by 68
  12. 2021Exponential Graph is Provably Efficient for Decentralized Deep Trainingcited by 56
  13. 2021Consensus Control for Decentralized Deep Learningcited by 49
  14. 2016On Nonconvex Decentralized Gradient Descentcited by 42
  15. 2021DecentLaM: Decentralized Momentum SGD for Large-batch Deep Trainingcited by 40
  16. 2020Decentralized gradient methods: does topology matter?cited by 35
  17. 2024Fewer Truncations Improve Language Modelingcited by 35
  18. 2019DADAM: A Consensus-based Distributed Adaptive Gradient Method for Online Optimizationcited by 30
  19. 2020Is Network the Bottleneck of Distributed Training?cited by 20
  20. 2021On the Convergence of Decentralized Adaptive Gradient Methodscited by 17
  21. 2020Periodic Stochastic Gradient Descent with Momentum for Decentralized Trainingcited by 14
  22. 2023Beyond Exponential Graph: Communication-Efficient Topologies for Decentralized Learning via Finite-time Convergencecited by 9
  23. 2023DropCompute: simple and more robust distributed synchronous training via compute variance reductioncited by 4
  24. 2023Is Consensus Acceleration Possible in Decentralized Optimization over Slowly Time-Varying Networks?cited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Unifying Local Communications and Local Updates for LLM Pretrainingcited by 1
  2. 2026Accelerating Decentralized Optimization via Overlapping Local Stepscited by 0
  3. 2026Decentralized SGD with Controlled Disagreement Finds Flatter Minimacited by 0
  4. 2026LoDAdaC: a unified local training-based decentralized framework with adaptive gradients and compressed communicationcited by 0