# Torpor: GPU-Enabled Serverless Computing for Low-Latency, Resource-Efficient Inference

arXiv: https://arxiv.org/abs/2306.03622 · HTML: https://mlsanity.com/paper/2306.03622/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Jun 2023.

## Built on by (most cited first, 13 of 13)

- [ServerlessLLM: Low-Latency Serverless Inference for Large Language Models](https://mlsanity.com/paper/2401.14351/) (2024)
- [RollArt: Disaggregated Multi-Task Agentic RL Training at Scale](https://mlsanity.com/paper/2512.22560/) (2025)
- [Janus: Disaggregating Attention and Experts for Scalable MoE Inference](https://mlsanity.com/paper/2512.13525/) (2025)
- [WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Serving](https://mlsanity.com/paper/2512.09472/) (2025)
- [Adaptive GPU Resource Allocation for Multi-Agent Collaborative Reasoning in Serverless Environments](https://mlsanity.com/paper/2512.22149/) (2025)
- [RAP: Runtime Adaptive Pruning for LLM Inference](https://mlsanity.com/paper/2505.17138/) (2025)
- [ServerlessLoRA: Enabling Low-Latency Serverless Multi-LoRA Serving](https://mlsanity.com/paper/2505.14468/) (2025)
- [Serverless GPU Architecture for Enterprise HR Analytics: A Production-Scale BDaaS Implementation](https://mlsanity.com/paper/2510.19689/) (2025)
- [Scepsy: Serving Agentic Workflows Using Aggregate LLM Pipelines](https://mlsanity.com/paper/2604.15186/) (2026)
- [A Survey of Serverless Machine Learning Model Inference](https://mlsanity.com/paper/2311.13587/) (2023)
- [FlexNPU: Transparent NPU Virtualization for Dynamic LLM Prefill-Decode Co-location](https://mlsanity.com/paper/2606.04415/) (2026)
- [ServerlessT2I: Efficient Text-to-Image Workflow Serving on a Serverless Platform](https://mlsanity.com/paper/2607.26566/) (2026)
- [Fast Recovery for LLM Serving via Decoupled Device Memory Lifetime in Dynamo](https://mlsanity.com/paper/2609.25451/) (2026)

## Abstract (arXiv)

Serverless computing offers a compelling cloud model for online inference services. However, existing serverless platforms lack efficient support for GPUs, hindering their ability to deliver high-performance inference. In this paper, we present Torpor, a serverless platform for GPU-efficient, low-latency inference. To enable efficient sharing of a node's GPUs among numerous inference functions, Torpor maintains models in main memory and dynamically swaps them onto GPUs upon request arrivals (i.e., late binding with model swapping). Torpor uses various techniques, including asynchronous API redirection, GPU runtime sharing, pipelined model execution, and efficient GPU memory management, to minimize latency overhead caused by model swapping. Additionally, we design an interference-aware request scheduling algorithm that utilizes high-speed GPU interconnects to meet latency service-level objectives (SLOs) for individual inference functions. We have implemented Torpor and evaluated its performance in a production environment. Utilizing late binding and model swapping, Torpor can concurrently serve hundreds of inference functions on a worker node with 4 GPUs, while achieving latency performance comparable to native execution, where each model is cached exclusively on a GPU. Pilot deployment in a leading commercial serverless cloud shows that Torpor reduces the GPU provisioning cost by 70% and 65% for users and the platform, respectively.