mlsanityData to 7 Oct 2026

CD-SGD: Distributed Stochastic Gradient Descent with Compression and Delay Compensation

arXiv 2106.10796 ↗First posted 21 Jun 2021cs.LG cs.DCcited by 0

Not on the map. Why?

Abstract (arXiv)

Communication overhead is the key challenge for distributed training. Gradient compression is a widely used approach to reduce communication traffic. When combining with parallel communication mechanism method like pipeline, gradient compression technique can greatly alleviate the impact of communication overhead. However, there exists two problems of gradient compression technique to be solved. Firstly, gradient compression brings in extra computation cost, which will delay the next training iteration. Secondly, gradient compression usually leads to the decrease of convergence accuracy.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 29 of its 44 references that are in the map. First posted on arXiv 21 Jun 2021.

Builds on

29 of 44 references.

References · 29

  1. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  2. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  3. 2018Local SGD Converges Fast and Communicates Littlecited by 595
  4. 2018Rethinking the Value of Network Pruningcited by 595
  5. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 561
  6. 2017Large Batch Training of Convolutional Networkscited by 529
  7. 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 471
  8. 2019Error Feedback Fixes SignSGD and other Gradient Compression Schemescited by 306
  9. 2017TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learningcited by 253
  10. 2017Sparse Communication for Distributed Gradient Descentcited by 243
  11. 2018Don't Use Large Mini-Batches, Use Local SGDcited by 201
  12. 2015MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systemscited by 195
17 more
  1. 2018Stochastic Gradient Push for Distributed Deep Learningcited by 152
  2. 2016QSGD: Communication-Efficient SGD via Gradient Quantization and Encodingcited by 147
  3. 2018Scalable Methods for 8-bit Training of Neural Networkscited by 139
  4. 2019Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronizationcited by 106
  5. 2017On the convergence properties of a K-step averaging stochastic gradient descent algorithm for nonconvex optimizationcited by 91
  6. 2018Training and Inference with Integers in Deep Neural Networkscited by 84
  7. 2017Poseidon: An Efficient Communication Architecture for Distributed Deep Learning on GPU Clusterscited by 55
  8. 2018SparCML: High-Performance Sparse Communication for Machine Learningcited by 34
  9. 2017AdaComp : Adaptive Residual Gradient Compression for Data-Parallel Distributed Trainingcited by 33
  10. 2018Pipe-SGD: A Decentralized Pipelined SGD Framework for Distributed Deep Net Trainingcited by 28
  11. 2018TicTac: Accelerating Distributed Deep Learning with Communication Schedulingcited by 27
  12. 2019Large-Batch Training for LSTM and Beyondcited by 21
  13. 2019Optimizing Network Performance for Distributed DNN Training on GPU Clusters: ImageNet/AlexNet Training in 1.5 Minutescited by 13
  14. 2018MG-WFBP: Efficient Data Communication for Distributed Synchronous SGD Algorithmscited by 12
  15. 2019Taming Unbalanced Training Workloads in Deep Learning with Partial Collective Operationscited by 9
  16. 2019Layer-wise Adaptive Gradient Sparsification for Distributed Deep Learning with Convergence Guaranteescited by 8
  17. 2020OD-SGD: One-step Delay Stochastic Gradient Descent for Distributed Trainingcited by 1

Built on by

Not cited yet.