# A Hitchhiker's Guide On Distributed Training of Deep Neural Networks

arXiv: https://arxiv.org/abs/1810.11787 · HTML: https://mlsanity.com/paper/1810.11787/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Oct 2018.

## Built on by (most cited first, 7 of 7)

- [A Comprehensive Survey on Distributed Training of Graph Neural Networks](https://mlsanity.com/paper/2211.05368/) (2022)
- [The Evolution of Distributed Systems for Graph Neural Networks and their Origin in Graph Processing and Deep Learning: A Survey](https://mlsanity.com/paper/2305.13854/) (2023)
- [On-device Federated Learning with Flower](https://mlsanity.com/paper/2104.03042/) (2021)
- [An Invitation to Distributed Quantum Neural Networks](https://mlsanity.com/paper/2211.07056/) (2022)
- [Anomaly Detection through Unsupervised Federated Learning](https://mlsanity.com/paper/2209.04184/) (2022)
- [Acceleration for Deep Reinforcement Learning using Parallel and Distributed Computing: A Survey](https://mlsanity.com/paper/2411.05614/) (2024)
- [A Review on Building Blocks of Decentralized Artificial Intelligence](https://mlsanity.com/paper/2402.02885/) (2024)

## Abstract (arXiv)

Deep learning has led to tremendous advancements in the field of Artificial Intelligence. One caveat however is the substantial amount of compute needed to train these deep learning models. Training a benchmark dataset like ImageNet on a single machine with a modern GPU can take upto a week, distributing training on multiple machines has been observed to drastically bring this time down. Recent work has brought down ImageNet training time to a time as low as 4 minutes by using a cluster of 2048 GPUs. This paper surveys the various algorithms and techniques used to distribute training and presents the current state of the art for a modern distributed training framework. More specifically, we explore the synchronous and asynchronous variants of distributed Stochastic Gradient Descent, various All Reduce gradient aggregation strategies and best practices for obtaining higher throughout and lower latency over a cluster such as mixed precision training, large batch training and gradient compression.