# Benchmarking State-of-the-Art Deep Learning Software Tools

arXiv: https://arxiv.org/abs/1608.07249 · HTML: https://mlsanity.com/paper/1608.07249/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Aug 2016.

## Built on by (most cited first, 17 of 17)

- [Physics-constrained Deep Learning for Robust Inverse ECG Modeling](https://mlsanity.com/paper/2107.12780/) (2021)
- [Dissecting the Runtime Performance of the Training, Fine-tuning, and Inference of Large Language Models](https://mlsanity.com/paper/2311.03687/) (2023)
- [Towards Enhancing the Reproducibility of Deep Learning Bugs: An Empirical Study](https://mlsanity.com/paper/2401.03069/) (2024)
- [Exploring Effects of Computational Parameter Changes to Image Recognition Systems](https://mlsanity.com/paper/2211.00471/) (2022)
- [DeltaNN: Assessing the Impact of Computational Environment Parameters on the Performance of Image Recognition Models](https://mlsanity.com/paper/2306.06208/) (2023)
- [Beyond Accuracy: A Consolidated Tool for Visual Question Answering Benchmarking](https://mlsanity.com/paper/2110.05159/) (2021)
- [A Comparative Survey of PyTorch vs TensorFlow for Deep Learning: Usability, Performance, and Deployment Trade-offs](https://mlsanity.com/paper/2508.04035/) (2025)
- [Generating Reliable Process Event Streams and Time Series Data based on Neural Networks](https://mlsanity.com/paper/2103.05462/) (2021)
- [Data Science Methodologies: Current Challenges and Future Approaches](https://mlsanity.com/paper/2106.07287/) (2021)
- [Deep Learning Models on CPUs: A Methodology for Efficient Training](https://mlsanity.com/paper/2206.10034/) (2022)
- [An Overview of the Data-Loader Landscape: Comparative Performance Analysis](https://mlsanity.com/paper/2209.13705/) (2022)
- [A Generic Performance Model for Deep Learning in a Distributed Environment](https://mlsanity.com/paper/2305.11665/) (2023)
- [Unveiling the frontiers of deep learning: innovations shaping diverse domains](https://mlsanity.com/paper/2309.02712/) (2023)
- [Benchmark Analysis of Various Pre-trained Deep Learning Models on ASSIRA Cats and Dogs Dataset](https://mlsanity.com/paper/2401.04666/) (2024)
- [Improving the network traffic classification using the Packet Vision approach](https://mlsanity.com/paper/2412.19360/) (2024)
- [A Theoretical Framework for Adaptive Utility-Weighted Benchmarking](https://mlsanity.com/paper/2602.12356/) (2026)
- [Benchmarking Storage Systems for Machine Learning Workloads Using NIO Bench](https://mlsanity.com/paper/2609.05418/) (2026)

## Abstract (arXiv)

Deep learning has been shown as a successful machine learning method for a variety of tasks, and its popularity results in numerous open-source deep learning software tools. Training a deep network is usually a very time-consuming process. To address the computational challenge in deep learning, many tools exploit hardware features such as multi-core CPUs and many-core GPUs to shorten the training time. However, different tools exhibit different features and running performance when training different types of deep networks on different hardware platforms, which makes it difficult for end users to select an appropriate pair of software and hardware. In this paper, we aim to make a comparative study of the state-of-the-art GPU-accelerated deep learning software tools, including Caffe, CNTK, MXNet, TensorFlow, and Torch. We first benchmark the running performance of these tools with three popular types of neural networks on two CPU platforms and three GPU platforms. We then benchmark some distributed versions on multiple GPUs. Our contribution is two-fold. First, for end users of deep learning tools, our benchmarking results can serve as a guide to selecting appropriate hardware platforms and software tools. Second, for software developers of deep learning tools, our in-depth analysis points out possible future directions to further optimize the running performance.