# A Generic Performance Model for Deep Learning in a Distributed Environment

arXiv: https://arxiv.org/abs/2305.11665 · HTML: https://mlsanity.com/paper/2305.11665/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 49 references that are in the map.
- First posted on arXiv 19 May 2023.

## Builds on (most cited first, 9 of 9)

- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [A Survey of the Recent Architectures of Deep Convolutional Neural Networks](https://mlsanity.com/paper/1901.06032/) (2019)
- [Resnet in Resnet: Generalizing Residual Architectures](https://mlsanity.com/paper/1603.08029/) (2016)
- [Theano: A Python framework for fast computation of mathematical expressions](https://mlsanity.com/paper/1605.02688/) (2016)
- [Benchmarking State-of-the-Art Deep Learning Software Tools](https://mlsanity.com/paper/1608.07249/) (2016)
- [Places205-VGGNet Models for Scene Recognition](https://mlsanity.com/paper/1508.01667/) (2015)
- [Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs](https://mlsanity.com/paper/1711.05979/) (2017)
- [Crystal structure prediction of materials with high symmetry using differential evolution](https://mlsanity.com/paper/2104.09764/) (2021)
- [CIFAR-10 Image Classification Using Feature Ensembles](https://mlsanity.com/paper/2002.03846/) (2020)

## Abstract (arXiv)

Performance modelling of a deep learning application is essential to improve and quantify the efficiency of the model framework. However, existing performance models are mostly case-specific, with limited capability for the new deep learning frameworks/applications. In this paper, we propose a generic performance model of an application in a distributed environment with a generic expression of the application execution time that considers the influence of both intrinsic factors/operations (e.g. algorithmic parameters/internal operations) and extrinsic scaling factors (e.g. the number of processors, data chunks and batch size). We formulate it as a global optimization problem and solve it using regularization on a cost function and differential evolution algorithm to find the best-fit values of the constants in the generic expression to match the experimentally determined computation time. We have evaluated the proposed model on three deep learning frameworks (i.e., TensorFlow, MXnet, and Pytorch). The experimental results show that the proposed model can provide accurate performance predictions and interpretability. In addition, the proposed work can be applied to any distributed deep neural network without instrumenting the code and provides insight into the factors affecting performance and scalability.