# Distributed Training of Deep Neural Networks: Theoretical and Practical Limits of Parallel Scalability

arXiv: https://arxiv.org/abs/1609.06870 · HTML: https://mlsanity.com/paper/1609.06870/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Sep 2016.

## Built on by (most cited first, 6 of 6)

- [Federated Learning on Non-IID Data: A Survey](https://mlsanity.com/paper/2106.06843/) (2021)
- [Accelerating Neural Network Training with Distributed Asynchronous and Selective Optimization (DASO)](https://mlsanity.com/paper/2104.05588/) (2021)
- [EmbRace: Accelerating Sparse Communication for Distributed Training of NLP Neural Networks](https://mlsanity.com/paper/2110.09132/) (2021)
- [Energy Consumption in Parallel Neural Network Training](https://mlsanity.com/paper/2508.07706/) (2025)
- [Exploring shared memory architectures for end-to-end gigapixel deep learning](https://mlsanity.com/paper/2304.12149/) (2023)
- [LAYUP: Asynchronous decentralized gradient descent with LAYer-wise UPdates](https://mlsanity.com/paper/2410.05985/) (2024)

## Abstract (arXiv)

This paper presents a theoretical analysis and practical evaluation of the main bottlenecks towards a scalable distributed solution for the training of Deep Neuronal Networks (DNNs). The presented results show, that the current state of the art approach, using data-parallelized Stochastic Gradient Descent (SGD), is quickly turning into a vastly communication bound problem. In addition, we present simple but fixed theoretic constraints, preventing effective scaling of DNN training beyond only a few dozen nodes. This leads to poor scalability of DNN training in most practical scenarios.