# Beyond Inference: Performance Analysis of DNN Server Overheads for Computer Vision

arXiv: https://arxiv.org/abs/2403.12981 · HTML: https://mlsanity.com/paper/2403.12981/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 17 references that are in the map.
- First posted on arXiv 2 Mar 2024.

## Builds on (most cited first, 6 of 6)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings](https://mlsanity.com/paper/2304.01433/) (2023)
- [MLPerf Inference Benchmark](https://mlsanity.com/paper/1911.02549/) (2019)
- [Fathom: Reference Workloads for Modern Deep Learning Methods](https://mlsanity.com/paper/1608.06581/) (2016)
- [DLA: Compiler and FPGA Overlay for Neural Network Inference Acceleration](https://mlsanity.com/paper/1807.06434/) (2018)
- [AI Tax: The Hidden Cost of AI Data Center Applications](https://mlsanity.com/paper/2007.10571/) (2020)

## Built on by (most cited first, 1 of 1)

- [How Much Does It Cost to Answer My Question? Benchmarking Cloud VLM-based VQA Systems](https://mlsanity.com/paper/2608.07861/) (2026)

## Abstract (arXiv)

Deep neural network (DNN) inference has become an important part of many data-center workloads. This has prompted focused efforts to design ever-faster deep learning accelerators such as GPUs and TPUs. However, an end-to-end DNN-based vision application contains more than just DNN inference, including input decompression, resizing, sampling, normalization, and data transfer. In this paper, we perform a thorough evaluation of computer vision inference requests performed on a throughput-optimized serving system. We quantify the performance impact of server overheads such as data movement, preprocessing, and message brokers between two DNNs producing outputs at different rates. Our empirical analysis encompasses many computer vision tasks including image classification, segmentation, detection, depth-estimation, and more complex processing pipelines with multiple DNNs. Our results consistently demonstrate that end-to-end application performance can easily be dominated by data processing and data movement functions (up to 56% of end-to-end latency in a medium-sized image, and ~ 80% impact on system throughput in a large image), even though these functions have been conventionally overlooked in deep learning system design. Our work identifies important performance bottlenecks in different application scenarios, achieves 2.25× better throughput compared to prior work, and paves the way for more holistic deep learning system design.