mlsanityData to 14 Oct 2026

On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization

arXiv 1905.03817 ↗First posted 9 May 2019math.OC cs.LGcited by 180

Not on the map. Why?

Abstract (arXiv)

Recent developments on large-scale distributed machine learning applications, e.g., deep neural networks, benefit enormously from the advances in distributed non-convex optimization techniques, e.g., distributed Stochastic Gradient Descent (SGD). A series of recent works study the linear speedup property of distributed SGD variants with reduced communication. The linear speedup property enable us to scale out the computing capability by adding more computing nodes into our system. The reduced communication complexity is desirable since communication overhead is often the performance bottleneck in distributed systems. Recently, momentum methods are more and more widely adopted in training machine learning models and can often converge faster and generalize better. For example, many practitioners use distributed SGD with momentum to train deep neural networks with big data. However, it remains unclear whether any distributed momentum SGD possesses the same linear speedup property as distributed SGD and has reduced communication complexity. This paper fills the gap by considering a distributed communication efficient momentum SGD method and proving its linear speedup property.

Key facts. Cited by 180 papers in the map (data through 14 Oct 2026). First posted on arXiv 9 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 41; 2022: 49; 2023: 35; 2024: 24; 2025: 15; 2026: 16.

180 papers.

Sort:

By area:

  1. 2021A Field Guide to Federated Optimizationcited by 315
  2. 2021Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learningcited by 193
  3. 2021Federated Learning with Buffered Asynchronous Aggregationcited by 190
  4. 2022Adam Can Converge Without Any Modification On Update Rulescited by 114
  5. 2023Stochastic Controlled Averaging for Federated Learning with Communication Compressioncited by 101
  6. 2021Fast Federated Learning in the Presence of Arbitrary Device Unavailabilitycited by 72
  7. 2021Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Datacited by 69
  8. 2021Tackling System and Statistical Heterogeneity for Federated Learning with Adaptive Client Samplingcited by 66
  9. 2024AdaptSFL: Adaptive Split Federated Learning in Resource-constrained Edge Networkscited by 64
  10. 2021What Do We Mean by Generalization in Federated Learning?cited by 62
  11. 2021Exponential Graph is Provably Efficient for Decentralized Deep Trainingcited by 56
  12. 2021Decentralized Federated Learning: Balancing Communication and Computing Costscited by 55
38 more
  1. 2021A Novel Convergence Analysis for Algorithms of the Adam Familycited by 50
  2. 2022FedFA: Federated Learning with Feature Anchors to Align Features and Classifiers for Heterogeneous Datacited by 50
  3. 2023FedSpeed: Larger Local Interval, Less Communication Round, and Higher Generalization Accuracycited by 49
  4. 2022A Unified Analysis of Federated Learning with Arbitrary Client Participationcited by 45
  5. 2021STEM: A Stochastic Two-Sided Momentum Algorithm Achieving Near-Optimal Sample and Communication Complexities for Federated Learningcited by 43
  6. 2021Anarchic Federated Learningcited by 42
  7. 2022Closing the Generalization Gap of Cross-silo Federated Medical Image Segmentationcited by 42
  8. 2021DecentLaM: Decentralized Momentum SGD for Large-batch Deep Trainingcited by 40
  9. 2021Sharp Bounds for Federated Averaging (Local SGD) and Continuous Perspectivecited by 38
  10. 2022On the Unreasonable Effectiveness of Federated Averaging with Heterogeneous Datacited by 38
  11. 2023Stochastic Gradient Descent under Markovian Sampling Schemescited by 38
  12. 2023Momentum Benefits Non-IID Federated Learning Simply and Provablycited by 38
  13. 2024Hierarchical Split Federated Learning: Convergence Analysis and System Optimizationcited by 38
  14. 2022Federated Minimax Optimization: Improved Convergence Analyses and Algorithmscited by 37
  15. 2022Faster Adaptive Federated Learningcited by 35
  16. 2021Cross-Gradient Aggregation for Decentralized Learning from Non-IID datacited by 33
  17. 2022Lower Bounds and Nearly Optimal Algorithms in Distributed Learning with Communication Compressioncited by 33
  18. 2022Variance Reduced ProxSkip: Algorithm, Theory and Application to Federated Learningcited by 33
  19. 2021Local Adaptivity in Federated Learning: Convergence and Consistencycited by 31
  20. 2023On the Convergence of Federated Averaging with Cyclic Client Participationcited by 31
  21. 2023Momentum Provably Improves Error Feedback!cited by 31
  22. 2021Byzantine-robust Federated Learning through Collaborative Malicious Gradient Filteringcited by 30
  23. 2021Removing Data Heterogeneity Influence Enhances Network Topology Dependence of Decentralized SGDcited by 29
  24. 2021Towards Practical Adam: Non-Convexity, Convergence Theory, and Mini-Batch Accelerationcited by 25
  25. 2021Toward Communication Efficient Adaptive Gradient Methodcited by 25
  26. 2022Revisiting Optimal Convergence Rate for Smooth and Non-convex Stochastic Decentralized Optimizationcited by 23
  27. 2023Efficient Federated Learning via Local Adaptive Amended Optimizer with Linear Speedupcited by 23
  28. 2021Does Momentum Change the Implicit Regularization on Separable Data?cited by 22
  29. 2021Federated Deep AUC Maximization for Heterogeneous Data with a Constant Communication Complexitycited by 21
  30. 2022AdaBest: Minimizing Client Drift in Federated Learning via Adaptive Bias Estimationcited by 21
  31. 2022A Communication-Efficient Distributed Gradient Clipping Algorithm for Training Deep Neural Networkscited by 20
  32. 2022Momentum Tracking: Momentum Acceleration for Decentralized Deep Learning on Heterogeneous Datacited by 20
  33. 2022On Distributed Adaptive Optimization with Gradient Compressioncited by 19
  34. 2022Scalable and Low-Latency Federated Learning with Cooperative Mobile Edge Networkingcited by 18
  35. 2022GradSkip: Communication-Accelerated Local Gradient Methods with Better Computational Complexitycited by 18
  36. 2023Gradient Descent with Linearly Correlated Noise: Theory and Applications to Differential Privacycited by 18
  37. 2023The Marginal Value of Momentum for Small Learning Rate SGDcited by 18
  38. 2021Adversarial training in communication constrained federated learningcited by 17

50 most cited of 180 All 180 (JSON)