mlsanityData to 7 Oct 2026

Stochastic Gradient Push for Distributed Deep Learning

arXiv 1811.10792 ↗First posted 27 Nov 2018cs.LG cs.AI cs.DCcited by 152

Not on the map. Why?

Abstract (arXiv)

Distributed data-parallel algorithms aim to accelerate the training of deep neural networks by parallelizing the computation of large mini-batch gradient updates across multiple nodes. Approaches that synchronize nodes using exact distributed averaging (e.g., via AllReduce) are sensitive to stragglers and communication delays. The PushSum gossip algorithm is robust to these issues, but only performs approximate distributed averaging. This paper studies Stochastic Gradient Push (SGP), which combines PushSum with stochastic gradient updates. We prove that SGP converges to a stationary point of smooth, non-convex objectives at the same sub-linear rate as SGD, and that all nodes achieve consensus. We empirically validate the performance of SGP on image classification (ResNet-50, ImageNet) and machine translation (Transformer, WMT'16 En-De) workloads. Our code will be made publicly available.

Key facts. Cited by 152 papers in the map (data through 7 Oct 2026). First posted on arXiv 27 Nov 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 32; 2022: 29; 2023: 27; 2024: 21; 2025: 20; 2026: 23.

152 papers.

Sort:

By area:

  1. 2023Decentralized Federated Learning: A Survey and Perspectivecited by 86
  2. 2022Sharper Convergence Guarantees for Asynchronous SGD for Distributed and Federated Learningcited by 76
  3. 2021Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Datacited by 69
  4. 2021Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelinescited by 68
  5. 2022An Improved Analysis of Gradient Tracking for Decentralized Machine Learningcited by 68
  6. 2021From Distributed Machine Learning to Federated Learning: A Surveycited by 65
  7. 2022Asynchronous SGD Beats Minibatch SGD Under Arbitrary Delayscited by 61
  8. 2021Distributed Deep Learning in Open Collaborationscited by 58
  9. 2021Decentralized Federated Learning through Proxy Model Sharingcited by 56
  10. 2021Exponential Graph is Provably Efficient for Decentralized Deep Trainingcited by 55
  11. 2022FedAvg with Fine Tuning: Local Updates Lead to Representation Learningcited by 55
  12. 2021Consensus Control for Decentralized Deep Learningcited by 49
38 more
  1. 2021Stochastic Gradient Descent-Ascent and Consensus Optimization for Smooth Games: Convergence Analysis under Expected Co-coercivitycited by 43
  2. 2021DecentLaM: Decentralized Momentum SGD for Large-batch Deep Trainingcited by 40
  3. 2021RelaySum for Decentralized Deep Learning on Heterogeneous Datacited by 39
  4. 2022Topology-aware Generalization of Decentralized SGDcited by 37
  5. 2021Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devicescited by 36
  6. 2022On the Unreasonable Effectiveness of Federated Averaging with Heterogeneous Datacited by 36
  7. 2021Decentralized Riemannian Gradient Descent on the Stiefel Manifoldcited by 33
  8. 2022Lower Bounds and Nearly Optimal Algorithms in Distributed Learning with Communication Compressioncited by 33
  9. 2021Cross-Gradient Aggregation for Decentralized Learning from Non-IID datacited by 32
  10. 2022Refined Convergence and Topology Learning for Decentralized SGD with Heterogeneous Datacited by 31
  11. 2022Byzantine-Robust Decentralized Learning via ClippedGossipcited by 29
  12. 2021Removing Data Heterogeneity Influence Enhances Network Topology Dependence of Decentralized SGDcited by 28
  13. 2022Beyond spectral gap: The role of the topology in decentralized learningcited by 27
  14. 2021Accelerating Gossip SGD with Periodic Global Averagingcited by 25
  15. 2021Decentralized Local Stochastic Extra-Gradient for Variational Inequalitiescited by 25
  16. 2022Communication-Efficient Topologies for Decentralized Learning with O(1) Consensus Ratecited by 25
  17. 2021Decentralized federated learning of deep neural networks on non-iid datacited by 23
  18. 2022Revisiting Optimal Convergence Rate for Smooth and Non-convex Stochastic Decentralized Optimizationcited by 22
  19. 2021A Hybrid Variance-Reduced Method for Decentralized Stochastic Non-Convex Optimizationcited by 21
  20. 2023Epidemic Learning: Boosting Decentralized Learning with Randomized Communicationcited by 21
  21. 2024Overcoming Data and Model Heterogeneities in Decentralized Federated Learning via Synthetic Anchorscited by 21
  22. 2022Momentum Tracking: Momentum Acceleration for Decentralized Deep Learning on Heterogeneous Datacited by 20
  23. 2021Trade-offs of Local SGD at Scale: An Empirical Studycited by 19
  24. 2021Distributed Learning Systems with First-order Methodscited by 17
  25. 2021On the Convergence of Decentralized Adaptive Gradient Methodscited by 17
  26. 2024CO2: Efficient Distributed Training with Full Communication-Computation Overlapcited by 17
  27. 2021DESTRESS: Computation-Optimal and Communication-Efficient Decentralized Nonconvex Finite-Sum Optimizationcited by 16
  28. 2021BlueFog: Make Decentralized Algorithms Practical for Optimization and Deep Learningcited by 16
  29. 2022Semi-Decentralized Federated Learning with Collaborative Relayingcited by 16
  30. 2023Decentralized SGD and Average-direction SAM are Asymptotically Equivalentcited by 15
  31. 2023AEDFL: Efficient Asynchronous Decentralized Federated Learning with Heterogeneous Devicescited by 15
  32. 2021Improving the Transient Times for Distributed Stochastic Gradient Methodscited by 13
  33. 2024Stochastic Polyak Step-sizes and Momentum: Convergence Guarantees and Practical Performancecited by 13
  34. 2022Data-heterogeneity-aware Mixing for Decentralized Learningcited by 12
  35. 2023Global Update Tracking: A Decentralized Learning Algorithm for Heterogeneous Datacited by 12
  36. 2023DIGEST: Fast and Communication Efficient Decentralized Learning with Local Updatescited by 12
  37. 2021Secure Distributed Training at Scalecited by 11
  38. 2022Tackling Data Heterogeneity: A New Unified Framework for Decentralized SGD with Sample-induced Topologycited by 10

50 most cited of 152 All 152 (JSON)