# Deep Learning Inference Frameworks Benchmark

arXiv: https://arxiv.org/abs/2210.04323 · HTML: https://mlsanity.com/paper/2210.04323/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 21 references that are in the map.
- First posted on arXiv 9 Oct 2022.

## Builds on (most cited first, 11 of 11)

- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [BinaryConnect: Training Deep Neural Networks with binary weights during propagations](https://mlsanity.com/paper/1511.00363/) (2015)
- [Model compression via distillation and quantization](https://mlsanity.com/paper/1802.05668/) (2018)
- [MLPerf Inference Benchmark](https://mlsanity.com/paper/1911.02549/) (2019)
- [Clipper: A Low-Latency Online Prediction Serving System](https://mlsanity.com/paper/1612.03079/) (2016)
- [TVM: An Automated End-to-End Optimizing Compiler for Deep Learning](https://mlsanity.com/paper/1802.04799/) (2018)
- [Scalable Model Compression by Entropy Penalized Reparameterization](https://mlsanity.com/paper/1906.06624/) (2019)
- [Analysis of DAWNBench, a Time-to-Accuracy Machine Learning Performance Benchmark](https://mlsanity.com/paper/1806.01427/) (2018)
- [Breaking Batch Normalization for better explainability of Deep Neural Networks through Layer-wise Relevance Propagation](https://mlsanity.com/paper/2002.11018/) (2020)
- [AI Matrix: A Deep Learning Benchmark for Alibaba Data Centers](https://mlsanity.com/paper/1909.10562/) (2019)
- [An efficient and flexible inference system for serving heterogeneous ensembles of deep neural networks](https://mlsanity.com/paper/2208.14049/) (2022)

## Built on by (most cited first, 2 of 2)

- [Small Object Detection with YOLO: A Performance Analysis Across Model Versions and Hardware](https://mlsanity.com/paper/2504.09900/) (2025)
- [KD-PINN: Knowledge-Distilled PINNs for ultra-low-latency real-time neural PDE solvers](https://mlsanity.com/paper/2512.13336/) (2025)

## Abstract (arXiv)

Deep learning (DL) has been widely adopted those last years but they are computing-intensive method. Therefore, scientists proposed diverse optimization to accelerate their predictions for end-user applications. However, no single inference framework currently dominates in terms of performance. This paper takes a holistic approach to conduct an empirical comparison and analysis of four representative DL inference frameworks. First, given a selection of CPU-GPU configurations, we show that for a specific DL framework, different configurations of its settings may have a significant impact on the prediction speed, memory, and computing power. Second, to the best of our knowledge, this study is the first to identify the opportunities for accelerating the ensemble of co-localized models in the same GPU. This measurement study provides an in-depth empirical comparison and analysis of four representative DL frameworks and offers practical guidance for service providers to deploy and deliver DL predictions.