# Faster Distributed Deep Net Training: Computation and Communication Decoupled Stochastic Gradient Descent

arXiv: https://arxiv.org/abs/1906.12043 · HTML: https://mlsanity.com/paper/1906.12043/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jun 2019.

## Built on by (most cited first, 7 of 7)

- [ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Training](https://mlsanity.com/paper/2406.02613/) (2024)
- [Joint Local Relational Augmentation and Global Nash Equilibrium for Federated Learning with Non-IID Data](https://mlsanity.com/paper/2308.11646/) (2023)
- [FedCos: A Scene-adaptive Federated Optimization Enhancement for Performance Improvement](https://mlsanity.com/paper/2204.03174/) (2022)
- [CC-FedAvg: Computationally Customized Federated Averaging](https://mlsanity.com/paper/2212.13679/) (2022)
- [Federated Learning via Consensus Mechanism on Heterogeneous Data: A New Perspective on Convergence](https://mlsanity.com/paper/2311.12358/) (2023)
- [FedEx: Expediting Federated Learning over Heterogeneous Mobile Devices by Overlapping and Participant Selection](https://mlsanity.com/paper/2407.00943/) (2024)
- [Accelerating Decentralized Optimization via Overlapping Local Steps](https://mlsanity.com/paper/2601.01493/) (2026)

## Abstract (arXiv)

With the increase in the amount of data and the expansion of model scale, distributed parallel training becomes an important and successful technique to address the optimization challenges. Nevertheless, although distributed stochastic gradient descent (SGD) algorithms can achieve a linear iteration speedup, they are limited significantly in practice by the communication cost, making it difficult to achieve a linear time speedup. In this paper, we propose a computation and communication decoupled stochastic gradient descent (CoCoD-SGD) algorithm to run computation and communication in parallel to reduce the communication cost. We prove that CoCoD-SGD has a linear iteration speedup with respect to the total computation capability of the hardware resources. In addition, it has a lower communication complexity and better time speedup comparing with traditional distributed SGD algorithms. Experiments on deep neural network training demonstrate the significant improvements of CoCoD-SGD: when training ResNet18 and VGG16 with 16 Geforce GTX 1080Ti GPUs, CoCoD-SGD is up to 2-3× faster than traditional synchronous SGD.