mlsanityData to 7 Oct 2026

Distributed Learning Systems with First-order Methods

arXiv 2104.05245 ↗First posted 12 Apr 2021cs.LG cs.DB cs.DCcited by 17

Not on the map. Why?

Abstract (arXiv)

Scalable and efficient distributed learning is one of the main driving forces behind the recent rapid advancement of machine learning and artificial intelligence. One prominent feature of this topic is that recent progresses have been made by researchers in two communities: (1) the system community such as database, data management, and distributed systems, and (2) the machine learning and mathematical optimization community. The interaction and knowledge sharing between these two communities has led to the rapid development of new distributed learning systems and theory. In this work, we hope to provide a brief introduction of some distributed learning techniques that have recently been developed, namely lossy communication compression (e.g., quantization and sparsification), asynchronous communication, and decentralized communication. One special focus in this work is on making sure that it can be easily understood by researchers in both communities -- On the system side, we rely on a simplified system model hiding many system details that are not necessary for the intuition behind the system speedups; while, on the theory side, we rely on minimal assumptions and significantly simplify the proof of some recent work to achieve comparable results.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 106 of its 142 references that are in the map. First posted on arXiv 12 Apr 2021.

Builds on

106 of 142 references.

References · 106

  1. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  2. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  3. 2018Byzantine-Robust Distributed Learning: Towards Optimal Statistical Ratescited by 698
  4. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 561
  5. 2017Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descentcited by 496
  6. 2018Sparsified SGD with Memorycited by 373
  7. 2015Deep Learning with Limited Numerical Precisioncited by 368
  8. 2020A Unified Theory of Decentralized SGD with Changing Topology and Local Updatescited by 361
  9. 2018Not All Samples Are Created Equal: Deep Learning with Importance Samplingcited by 323
  10. 2019The Non-IID Data Quagmire of Decentralized Machine Learningcited by 303
  11. 2017TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learningcited by 253
  12. 2018The Convergence of Sparsified Gradient Methodscited by 228
38 more
  1. 2019Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communicationcited by 218
  2. 2017Gradient Sparsification for Communication-Efficient Distributed Optimizationcited by 199
  3. 2018Beyond Data and Model Parallelism for Deep Neural Networkscited by 177
  4. 2018Training Deep Neural Networks with 8-bit Floating Point Numberscited by 163
  5. 2014EXTRA: An Exact First-Order Algorithm for Decentralized Consensus Optimizationcited by 160
  6. 2017Asynchronous Decentralized Parallel Stochastic Gradient Descentcited by 159
  7. 2013On the Convergence of Decentralized Gradient Descentcited by 158
  8. 2018cpSGD: Communication-efficient and differentially-private distributed SGDcited by 157
  9. 2018Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysiscited by 154
  10. 2018Stochastic Gradient Push for Distributed Deep Learningcited by 152
  11. 2016QSGD: Communication-Efficient SGD via Gradient Quantization and Encodingcited by 147
  12. 2016Stochastic Variance Reduction for Nonconvex Optimizationcited by 143
  13. 2013Mini-batch Stochastic Approximation Methods for Nonconvex Stochastic Composite Optimizationcited by 142
  14. 2018D2: Decentralized Training over Decentralized Datacited by 140
  15. 2018Scalable Methods for 8-bit Training of Neural Networkscited by 139
  16. 2018ATOMO: Communication-efficient Learning via Atomic Sparsificationcited by 135
  17. 2018Deep Learning in Mobile and Wireless Networking: A Surveycited by 117
  18. 2015Asynchronous Parallel Stochastic Gradient for Nonconvex Optimizationcited by 115
  19. 2018Zeroth-Order Stochastic Variance Reduction for Nonconvex Optimizationcited by 109
  20. 2016Asynchronous Stochastic Gradient Descent with Delay Compensationcited by 98
  21. 2019DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compressioncited by 98
  22. 2018SGD and Hogwild! Convergence Without the Bounded Gradients Assumptioncited by 94
  23. 2014Stochastic Optimization with Importance Samplingcited by 91
  24. 2014Deep learning with Elastic Averaging SGDcited by 90
  25. 2018Communication Compression for Decentralized Trainingcited by 87
  26. 2018Byzantine Stochastic Gradient Descentcited by 81
  27. 2018Error Compensated Quantized SGD and its Applications to Large-scale Distributed Optimizationcited by 76
  28. 2018Adaptive Communication Strategies to Achieve the Best Error-Runtime Trade-off in Local-Update SGDcited by 76
  29. 2018DRACO: Byzantine-resilient Distributed Training via Redundant Gradientscited by 74
  30. 2016Variance Reduction for Faster Non-Convex Optimizationcited by 72
  31. 2018Random Shuffling Beats SGD after Finite Epochscited by 66
  32. 2019SGD without Replacement: Sharper Rates for General Smooth Convex Functionscited by 59
  33. 2016Stochastic Variance Reduction Methods for Saddle-Point Problemscited by 57
  34. 2017A decentralized proximal-gradient method with network independent step-sizes and separated convergence ratescited by 51
  35. 2019Improved Zeroth-Order Variance Reduced Algorithms and Analysis for Nonconvex Optimizationcited by 47
  36. 2016A Comprehensive Linear Speedup Analysis for Asynchronous Stochastic Parallel Optimization from Zeroth-Order to First-Ordercited by 43
  37. 2016Variance-Reduced and Projection-Free Stochastic Optimizationcited by 40
  38. 2015Variance Reduced Stochastic Gradient Descent with Neighborscited by 39

50 most cited of 106 All 106 (JSON)

Built on by

Cited by year

Citing papers per year: 2021: 6; 2022: 4; 2023: 3; 2024: 3; 2025: 1.

17 papers.

Sort:

  1. 2022Parallel Rectangle Flip Attack: A Query-based Black-box Attack against Object Detectioncited by 79
  2. 2021Quantized Federated Learning under Transmission Delay and Outage Constraintscited by 37
  3. 2022Fine-tuning Language Models over Slow Networks using Activation Compression with Guaranteescited by 29
  4. 2021Persia: An Open, Hybrid System Scaling Deep Learning-based Recommenders up to 100 Trillion Parameterscited by 17
  5. 2021Towards Demystifying Serverless Machine Learning Trainingcited by 16
  6. 2022Data Debugging with Shapley Importance over End-to-End Machine Learning Pipelinescited by 16
  7. 2021Basis Matters: Better Communication-Efficient Second Order Methods for Federated Learningcited by 14
  8. 2021BAGUA: Scaling up Distributed Learning with System Relaxationscited by 13
  9. 2023Quantized Distributed Training of Large Models with Convergence Guaranteescited by 13
  10. 2021Compressed Gradient Tracking for Decentralized Optimization Over General Directed Networkscited by 10
  11. 2023CEDAS: A Compressed Decentralized Stochastic Gradient Method with Improved Convergencecited by 9
  12. 2024PIM-Opt: Demystifying Distributed Optimization Algorithms on a Real-World Processing-In-Memory Systemcited by 5
5 more
  1. 2024Ordered Momentum for Asynchronous SGDcited by 5
  2. 2022Stochastic Gradient Descent without Full Data Shufflecited by 2
  3. 2024SignSGD with Federated Votingcited by 2
  4. 2023Plug In and Learn: Federated Intelligence over a Smart Grid of Modelscited by 1
  5. 2025L2T-Tune:LLM-Guided Hybrid Database Tuning with LHS and TD3cited by 1