# Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs

arXiv: https://arxiv.org/abs/1711.05979 · HTML: https://mlsanity.com/paper/1711.05979/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Nov 2017.

## Built on by (most cited first, 13 of 13)

- [Distributed Deep Learning in Open Collaborations](https://mlsanity.com/paper/2106.10207/) (2021)
- [An Efficient Statistical-based Gradient Compression Technique for Distributed Training Systems](https://mlsanity.com/paper/2101.10761/) (2021)
- [ResIST: Layer-Wise Decomposition of ResNets for Distributed Training](https://mlsanity.com/paper/2107.00961/) (2021)
- [Dissecting the Runtime Performance of the Training, Fine-tuning, and Inference of Large Language Models](https://mlsanity.com/paper/2311.03687/) (2023)
- [Exploring GPU-to-GPU Communication: Insights into Supercomputer Interconnects](https://mlsanity.com/paper/2408.14090/) (2024)
- [Accelerating Distributed K-FAC with Smart Parallelism of Computing and Communication Tasks](https://mlsanity.com/paper/2107.06533/) (2021)
- [Sync-Switch: Hybrid Parameter Synchronization for Distributed Deep Learning](https://mlsanity.com/paper/2104.08364/) (2021)
- [DeAR: Accelerating Distributed Deep Learning with Fine-Grained All-Reduce Pipelining](https://mlsanity.com/paper/2302.12445/) (2023)
- [EmbRace: Accelerating Sparse Communication for Distributed Training of NLP Neural Networks](https://mlsanity.com/paper/2110.09132/) (2021)
- [Communication-Efficient Distributed Deep Learning via Federated Dynamic Averaging](https://mlsanity.com/paper/2405.20988/) (2024)
- [FedSlate:A Federated Deep Reinforcement Learning Recommender System](https://mlsanity.com/paper/2409.14872/) (2024)
- [Improving the Performance of DNN-based Software Services using Automated Layer Caching](https://mlsanity.com/paper/2209.08625/) (2022)
- [A Generic Performance Model for Deep Learning in a Distributed Environment](https://mlsanity.com/paper/2305.11665/) (2023)

## Abstract (arXiv)

Deep learning frameworks have been widely deployed on GPU servers for deep learning applications in both academia and industry. In training deep neural networks (DNNs), there are many standard processes or algorithms, such as convolution and stochastic gradient descent (SGD), but the running performance of different frameworks might be different even running the same deep model on the same GPU hardware. In this study, we evaluate the running performance of four state-of-the-art distributed deep learning frameworks (i.e., Caffe-MPI, CNTK, MXNet, and TensorFlow) over single-GPU, multi-GPU, and multi-node environments. We first build performance models of standard processes in training DNNs with SGD, and then we benchmark the running performance of these frameworks with three popular convolutional neural networks (i.e., AlexNet, GoogleNet and ResNet-50), after that, we analyze what factors that result in the performance gap among these four frameworks. Through both analytical and experimental analysis, we identify bottlenecks and overheads which could be further optimized. The main contribution is that the proposed performance models and the analysis provide further optimization directions in both algorithmic design and system configuration.