# Automated Runtime-Aware Scheduling for Multi-Tenant DNN Inference on GPU

arXiv: https://arxiv.org/abs/2111.14255 · HTML: https://mlsanity.com/paper/2111.14255/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Nov 2021.

## Built on by (most cited first, 9 of 9)

- [Your Compiler is Backdooring Your Model: Understanding and Exploiting Compilation Inconsistency Vulnerabilities in Deep Learning Compilers](https://mlsanity.com/paper/2509.11173/) (2025)
- [Boosting DNN Cold Inference on Edge Devices](https://mlsanity.com/paper/2206.07446/) (2022)
- [Miriam: Exploiting Elastic Kernels for Real-time Multi-DNN Inference on Edge GPU](https://mlsanity.com/paper/2307.04339/) (2023)
- [Tao: Re-Thinking DL-based Microarchitecture Simulation](https://mlsanity.com/paper/2404.10921/) (2024)
- [Hardware-Assisted Virtualization of Neural Processing Units for Cloud Platforms](https://mlsanity.com/paper/2408.04104/) (2024)
- [FedHQ: Hybrid Runtime Quantization for Federated Learning](https://mlsanity.com/paper/2505.11982/) (2025)
- [CARIn: Constraint-Aware and Responsive Inference on Heterogeneous Devices for Single- and Multi-DNN Workloads](https://mlsanity.com/paper/2409.01089/) (2024)
- [Dynamic DNNs and Runtime Management for Efficient Inference on Mobile/Embedded Devices](https://mlsanity.com/paper/2401.08965/) (2024)
- [MeanField Surrogate Modeling for Scalable Runtime Scheduling of Concurrent Heterogeneous AI Inference on Shared GPUs](https://mlsanity.com/paper/2609.02109/) (2026)

## Abstract (arXiv)

With the fast development of deep neural networks (DNNs), many real-world applications are adopting multiple models to conduct compound tasks, such as co-running classification, detection, and segmentation models on autonomous vehicles. Such multi-tenant DNN inference cases greatly exacerbate the computational complexity and call for comprehensive collaboration for graph-level operator scheduling, runtime-level resource awareness, as well as hardware scheduler support. However, the current scheduling support for such multi-tenant inference is still relatively backward. In this work, we propose a resource-aware scheduling framework for efficient multi-tenant DNN inference on GPU, which automatically coordinates DNN computing in different execution levels. Leveraging the unified scheduling intermediate representation and the automated ML-based searching algorithm, optimal schedules could be generated to wisely adjust model concurrency and interleave DNN model operators, maintaining a continuously balanced resource utilization across the entire inference process, and eventually improving the runtime efficiency. Experiments show that we could consistently achieve 1.3-1.7x speed-up, compared to regular DNN runtime libraries (e.g., CuDNN, TVM) and particular concurrent scheduling methods (e.g., NVIDIA Multi-Stream).