# Distributed Training of Deep Learning Models: A Taxonomic Perspective

arXiv: https://arxiv.org/abs/2007.03970 · HTML: https://mlsanity.com/paper/2007.03970/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Jul 2020.

## Built on by (most cited first, 25 of 25)

- [Randomness In Neural Network Training: Characterizing The Impact of Tooling](https://mlsanity.com/paper/2106.11872/) (2021)
- [Distributed Graph Neural Network Training: A Survey](https://mlsanity.com/paper/2211.00216/) (2022)
- [Merlin HugeCTR: GPU-accelerated Recommender System Training and Inference](https://mlsanity.com/paper/2210.08803/) (2022)
- [Pervasive AI for IoT applications: A Survey on Resource-efficient Distributed Artificial Intelligence](https://mlsanity.com/paper/2105.01798/) (2021)
- [A GPU-specialized Inference Parameter Server for Large-Scale Deep Recommendation Models](https://mlsanity.com/paper/2210.08804/) (2022)
- [Communication-Efficient Federated Learning for Neural Machine Translation](https://mlsanity.com/paper/2112.06135/) (2021)
- [COMSPLIT: A Communication-Aware Split Learning Design for Heterogeneous IoT Platforms](https://mlsanity.com/paper/2410.19375/) (2024)
- [PrivSGP-VR: Differentially Private Variance-Reduced Stochastic Gradient Push with Tight Utility Bounds](https://mlsanity.com/paper/2405.02638/) (2024)
- [Federated learning and next generation wireless communications: A survey on bidirectional relationship](https://mlsanity.com/paper/2110.07649/) (2021)
- [Multi-limb Split Learning for Tumor Classification on Vertically Distributed Data](https://mlsanity.com/paper/2301.11468/) (2023)
- [Robust Fully-Asynchronous Methods for Distributed Training over General Architecture](https://mlsanity.com/paper/2307.11617/) (2023)
- [Roulette: A Semantic Privacy-Preserving Device-Edge Collaborative Inference Framework for Deep Learning Classification Tasks](https://mlsanity.com/paper/2309.02820/) (2023)
- [Training on Fake Labels: Mitigating Label Leakage in Split Learning via Secure Dimension Transformation](https://mlsanity.com/paper/2410.09125/) (2024)
- [Efficient Split Learning LSTM Models for FPGA-based Edge IoT Devices](https://mlsanity.com/paper/2502.08692/) (2025)
- [DP-CSGP: Differentially Private Stochastic Gradient Push with Compressed Communication](https://mlsanity.com/paper/2512.13583/) (2025)
- [Align and Filter: Improving Performance in Asynchronous On-Policy RL](https://mlsanity.com/paper/2603.01365/) (2026)
- [Quantize Once, Train Fast: Allreduce-Compatible Compression with Provable Guarantees](https://mlsanity.com/paper/2305.18627/) (2023)
- [A Survey From Distributed Machine Learning to Distributed Deep Learning](https://mlsanity.com/paper/2307.05232/) (2023)
- [Weighted Sampled Split Learning (WSSL): Balancing Privacy, Robustness, and Fairness in Distributed Learning Environments](https://mlsanity.com/paper/2310.18479/) (2023)
- [The Landscape of Modern Machine Learning: A Review of Machine, Distributed and Federated Learning](https://mlsanity.com/paper/2312.03120/) (2023)
- [ReDistill: Residual Encoded Distillation for Peak Memory Reduction of CNNs](https://mlsanity.com/paper/2406.03744/) (2024)
- [Dyn-D2P: Dynamic Differentially Private Decentralized Learning with Provable Utility Guarantee](https://mlsanity.com/paper/2505.06651/) (2025)
- [Byzantine-Robust and Communication-Efficient Distributed Training: Compressive and Cyclic Gradient Coding](https://mlsanity.com/paper/2603.28780/) (2026)
- [Adaptive AI Task Partitioning and Safe Offloading in Heterogeneous Edge-Cloud Continuum](https://mlsanity.com/paper/2605.09623/) (2026)
- [VersaDB: A High-Performance AI Storage Database for Unifying Mutimodal Datasets](https://mlsanity.com/paper/2608.22795/) (2026)

## Abstract (arXiv)

Distributed deep learning systems (DDLS) train deep neural network models by utilizing the distributed resources of a cluster. Developers of DDLS are required to make many decisions to process their particular workloads in their chosen environment efficiently. The advent of GPU-based deep learning, the ever-increasing size of datasets and deep neural network models, in combination with the bandwidth constraints that exist in cluster environments require developers of DDLS to be innovative in order to train high quality models quickly. Comparing DDLS side-by-side is difficult due to their extensive feature lists and architectural deviations. We aim to shine some light on the fundamental principles that are at work when training deep neural networks in a cluster of independent machines by analyzing the general properties associated with training deep learning models and how such workloads can be distributed in a cluster to achieve collaborative model training. Thereby we provide an overview of the different techniques that are used by contemporary DDLS and discuss their influence and implications on the training process. To conceptualize and compare DDLS, we group different techniques into categories, thus establishing a taxonomy of distributed deep learning systems.