# A Survey of Multi-Tenant Deep Learning Inference on GPU

arXiv: https://arxiv.org/abs/2203.09040 · HTML: https://mlsanity.com/paper/2203.09040/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Mar 2022.

## Built on by (most cited first, 7 of 7)

- [FlexLLM: Token-Level Co-Serving of LLM Inference and Finetuning with SLO Guarantees](https://mlsanity.com/paper/2402.18789/) (2024)
- [Deep Learning Workload Scheduling in GPU Datacenters: Taxonomy, Challenges and Vision](https://mlsanity.com/paper/2205.11913/) (2022)
- [Fast Tree-Field Integrators: From Low Displacement Rank to Topological Transformers](https://mlsanity.com/paper/2406.15881/) (2024)
- [CARIn: Constraint-Aware and Responsive Inference on Heterogeneous Devices for Single- and Multi-DNN Workloads](https://mlsanity.com/paper/2409.01089/) (2024)
- [A Collaborative PIM Computing Optimization Framework for Multi-Tenant DNN](https://mlsanity.com/paper/2408.04812/) (2024)
- [CoLLM: Continuous Adaptation for SLO-Aware LLM Serving on Shared GPU Clusters](https://mlsanity.com/paper/2604.16400/) (2026)
- [Strait: Perceiving Priority and Interference in ML Inference Serving](https://mlsanity.com/paper/2604.28175/) (2026)

## Abstract (arXiv)

Deep Learning (DL) models have achieved superior performance. Meanwhile, computing hardware like NVIDIA GPUs also demonstrated strong computing scaling trends with 2x throughput and memory bandwidth for each generation. With such strong computing scaling of GPUs, multi-tenant deep learning inference by co-locating multiple DL models onto the same GPU becomes widely deployed to improve resource utilization, enhance serving throughput, reduce energy cost, etc. However, achieving efficient multi-tenant DL inference is challenging which requires thorough full-stack system optimization. This survey aims to summarize and categorize the emerging challenges and optimization opportunities for multi-tenant DL inference on GPU. By overviewing the entire optimization stack, summarizing the multi-tenant computing innovations, and elaborating the recent technological advances, we hope that this survey could shed light on new optimization perspectives and motivate novel works in future large-scale DL system optimization.