mlsanityData to 14 Oct 2026

A Survey of Serverless Machine Learning Model Inference

arXiv 2311.13587 ↗First posted 22 Nov 2023cs.DC cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Recent developments in Generative AI, Computer Vision, and Natural Language Processing have led to an increased integration of AI models into various products. This widespread adoption of AI requires significant efforts in deploying these models in production environments. When hosting machine learning models for real-time predictions, it is important to meet defined Service Level Objectives (SLOs), ensuring reliability, minimal downtime, and optimizing operational costs of the underlying infrastructure. Large machine learning models often demand GPU resources for efficient inference to meet SLOs. In the context of these trends, there is growing interest in hosting AI models in a serverless architecture while still providing GPU access for inference tasks. This survey aims to summarize and categorize the emerging challenges and optimization opportunities for large-scale deep learning serving systems. By providing a novel taxonomy and summarizing recent trends, we hope that this survey could shed light on new optimization perspectives and motivate novel works in large-scale deep learning serving systems.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 23 of its 36 references that are in the map. First posted on arXiv 22 Nov 2023.

Builds on

23 of 36 references.

References · 23

  1. 2020Serving DNNs like Clockwork: Performance Predictability from the Bottom Upcited by 120
  2. 2020LazyBatching: An SLA-aware Batching System for Cloud Machine Learning Inferencecited by 22
  3. 2023MAD Max Beyond Single-Node: Enabling Large Machine Learning Model Acceleration on Distributed Systemscited by 14
  4. 2023Torpor: GPU-Enabled Serverless Computing for Low-Latency, Resource-Efficient Inferencecited by 13
  5. 2018Dynamic Space-Time Scheduling for GPU Inferencecited by 11
  6. 2020Faasm: Lightweight Isolation for Efficient Stateful Serverless Computingcited by 7
  7. 2021Serving DNN Models with Multi-Instance GPUs: A Case of the Reconfigurable Machine Scheduling Problemcited by 7
  8. 2022Demand Layering for Real-Time DNN Inference with Minimized Memory Usagecited by 7
  9. 2023FaST-GShare: Enabling Efficient Spatio-Temporal GPU Sharing in Serverless Computing for Deep Learning Inferencecited by 6
  10. 2023A Framework for Fine-Grained Synchronization of Dependent GPU Kernelscited by 5
  11. 2018TrIMS: Transparent and Isolated Model Sharing for Low Latency Deep LearningInference in Function as a Service Environmentscited by 4
  12. 2019The OoO VLIW JIT Compiler for GPU Inferencecited by 4
11 more
  1. 2021Multi-model Machine Learning Inference Serving with GPU Spatial Partitioningcited by 4
  2. 2023SMDP-Based Dynamic Batching for Efficient Inference on GPU-Based Platformscited by 4
  3. 2020Optimizing Prediction Serving on Low-Latency Serverless Dataflowcited by 3
  4. 2020Serverless inferencing on Kubernetescited by 2
  5. 2022PARIS and ELSA: An Elastic Scheduling Algorithm for Reconfigurable Multi-GPU Inference Serverscited by 2
  6. 2023GPU-enabled Function-as-a-Service for Machine Learning Inferencecited by 2
  7. 2023D-STACK: High Throughput DNN Inference by Effective Multiplexing and Spatio-Temporal Scheduling of GPUscited by 2
  8. 2020Accelerating Multi-Model Inference by Merging DNNs of Different Weightscited by 1
  9. 2022MLProxy: SLA-Aware Reverse Proxy for Machine Learning Inference Serving on Serverless Computing Platformscited by 1
  10. 2022Kernel-as-a-Service: A Serverless Interface to GPUscited by 1
  11. 2023Throughput Maximization of DNN Inference: Batching or Multi-Tenancy?cited by 1

Built on by

Not cited yet.