# Dynamic Space-Time Scheduling for GPU Inference

arXiv: https://arxiv.org/abs/1901.00041 · HTML: https://mlsanity.com/paper/1901.00041/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Dec 2018.

## Built on by (most cited first, 11 of 11)

- [Boggart: Towards General-Purpose Acceleration of Retrospective Video Analytics](https://mlsanity.com/paper/2106.15315/) (2021)
- [Deep Learning Workload Scheduling in GPU Datacenters: Taxonomy, Challenges and Vision](https://mlsanity.com/paper/2205.11913/) (2022)
- [Miriam: Exploiting Elastic Kernels for Real-time Multi-DNN Inference on Edge GPU](https://mlsanity.com/paper/2307.04339/) (2023)
- [Multi-model Machine Learning Inference Serving with GPU Spatial Partitioning](https://mlsanity.com/paper/2109.01611/) (2021)
- [CascadeServe: Unlocking Model Cascades for Inference Serving](https://mlsanity.com/paper/2406.14424/) (2024)
- [Boosting LLM Serving through Spatial-Temporal GPU Resource Sharing](https://mlsanity.com/paper/2504.19516/) (2025)
- [Characterizing Concurrency Mechanisms for NVIDIA GPUs under Deep Learning Workloads](https://mlsanity.com/paper/2110.00459/) (2021)
- [A Study on the Intersection of GPU Utilization and CNN Inference](https://mlsanity.com/paper/2212.07936/) (2022)
- [A Survey of Serverless Machine Learning Model Inference](https://mlsanity.com/paper/2311.13587/) (2023)
- [Hydro: Adaptive Query Processing of ML Queries](https://mlsanity.com/paper/2403.14902/) (2024)
- [A Study of Skews, Imbalances, and Pathological Conditions in LLM Inference Deployment on GPU Clusters detectable from DPU](https://mlsanity.com/paper/2509.18114/) (2025)

## Abstract (arXiv)

Serving deep neural networks in latency critical interactive settings often requires GPU acceleration. However, the small batch sizes typical in online inference results in poor GPU utilization, a potential performance gap which GPU resource sharing can address. In this paper, we explore several techniques to leverage both temporal and spatial multiplexing to improve GPU utilization for deep learning inference workloads. We evaluate the performance trade-offs of each approach with respect to resource-efficiency, latency predictability, and isolation when compared with conventional batched inference. Our experimental analysis suggests up to a 5x potential for improved utilization through the exploration of more advanced spatial and temporal multiplexing strategies. Our preliminary prototype of a dynamic space-time scheduler demonstrates a 3.23x floating-point throughput increase over space-only multiplexing and a 7.73x increase over time-only multiplexing for convolutions, while also providing better isolation and latency predictability.