# A Survey of Serverless Machine Learning Model Inference

arXiv: https://arxiv.org/abs/2311.13587 · HTML: https://mlsanity.com/paper/2311.13587/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 36 references that are in the map.
- First posted on arXiv 22 Nov 2023.

## Builds on (most cited first, 23 of 23)

- [Serving DNNs like Clockwork: Performance Predictability from the Bottom Up](https://mlsanity.com/paper/2006.02464/) (2020)
- [LazyBatching: An SLA-aware Batching System for Cloud Machine Learning Inference](https://mlsanity.com/paper/2010.13103/) (2020)
- [MAD Max Beyond Single-Node: Enabling Large Machine Learning Model Acceleration on Distributed Systems](https://mlsanity.com/paper/2310.02784/) (2023)
- [Torpor: GPU-Enabled Serverless Computing for Low-Latency, Resource-Efficient Inference](https://mlsanity.com/paper/2306.03622/) (2023)
- [Dynamic Space-Time Scheduling for GPU Inference](https://mlsanity.com/paper/1901.00041/) (2018)
- [Faasm: Lightweight Isolation for Efficient Stateful Serverless Computing](https://mlsanity.com/paper/2002.09344/) (2020)
- [Serving DNN Models with Multi-Instance GPUs: A Case of the Reconfigurable Machine Scheduling Problem](https://mlsanity.com/paper/2109.11067/) (2021)
- [Demand Layering for Real-Time DNN Inference with Minimized Memory Usage](https://mlsanity.com/paper/2210.04024/) (2022)
- [FaST-GShare: Enabling Efficient Spatio-Temporal GPU Sharing in Serverless Computing for Deep Learning Inference](https://mlsanity.com/paper/2309.00558/) (2023)
- [A Framework for Fine-Grained Synchronization of Dependent GPU Kernels](https://mlsanity.com/paper/2305.13450/) (2023)
- [TrIMS: Transparent and Isolated Model Sharing for Low Latency Deep LearningInference in Function as a Service Environments](https://mlsanity.com/paper/1811.09732/) (2018)
- [The OoO VLIW JIT Compiler for GPU Inference](https://mlsanity.com/paper/1901.10008/) (2019)
- [Multi-model Machine Learning Inference Serving with GPU Spatial Partitioning](https://mlsanity.com/paper/2109.01611/) (2021)
- [SMDP-Based Dynamic Batching for Efficient Inference on GPU-Based Platforms](https://mlsanity.com/paper/2301.12865/) (2023)
- [Optimizing Prediction Serving on Low-Latency Serverless Dataflow](https://mlsanity.com/paper/2007.05832/) (2020)
- [Serverless inferencing on Kubernetes](https://mlsanity.com/paper/2007.07366/) (2020)
- [PARIS and ELSA: An Elastic Scheduling Algorithm for Reconfigurable Multi-GPU Inference Servers](https://mlsanity.com/paper/2202.13481/) (2022)
- [GPU-enabled Function-as-a-Service for Machine Learning Inference](https://mlsanity.com/paper/2303.05601/) (2023)
- [D-STACK: High Throughput DNN Inference by Effective Multiplexing and Spatio-Temporal Scheduling of GPUs](https://mlsanity.com/paper/2304.13541/) (2023)
- [Accelerating Multi-Model Inference by Merging DNNs of Different Weights](https://mlsanity.com/paper/2009.13062/) (2020)
- [MLProxy: SLA-Aware Reverse Proxy for Machine Learning Inference Serving on Serverless Computing Platforms](https://mlsanity.com/paper/2202.11243/) (2022)
- [Kernel-as-a-Service: A Serverless Interface to GPUs](https://mlsanity.com/paper/2212.08146/) (2022)
- [Throughput Maximization of DNN Inference: Batching or Multi-Tenancy?](https://mlsanity.com/paper/2308.13803/) (2023)

## Abstract (arXiv)

Recent developments in Generative AI, Computer Vision, and Natural Language Processing have led to an increased integration of AI models into various products. This widespread adoption of AI requires significant efforts in deploying these models in production environments. When hosting machine learning models for real-time predictions, it is important to meet defined Service Level Objectives (SLOs), ensuring reliability, minimal downtime, and optimizing operational costs of the underlying infrastructure. Large machine learning models often demand GPU resources for efficient inference to meet SLOs. In the context of these trends, there is growing interest in hosting AI models in a serverless architecture while still providing GPU access for inference tasks. This survey aims to summarize and categorize the emerging challenges and optimization opportunities for large-scale deep learning serving systems. By providing a novel taxonomy and summarizing recent trends, we hope that this survey could shed light on new optimization perspectives and motivate novel works in large-scale deep learning serving systems.