# AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving

arXiv: https://arxiv.org/abs/2302.11665 · HTML: https://mlsanity.com/paper/2302.11665/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 57 references that are in the map.
- First posted on arXiv 22 Feb 2023.

## Builds on (most cited first, 10 of 10)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Efficiently Scaling Transformer Inference](https://mlsanity.com/paper/2211.05102/) (2022)
- [DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale](https://mlsanity.com/paper/2207.00032/) (2022)
- [GSPMD: General and Scalable Parallelization for ML Computation Graphs](https://mlsanity.com/paper/2105.04663/) (2021)
- [TensorFlow-Serving: Flexible, High-Performance ML Serving](https://mlsanity.com/paper/1712.06139/) (2017)
- [Robustness Challenges in Model Distillation and Pruning for Natural Language Understanding](https://mlsanity.com/paper/2110.08419/) (2021)

## Built on by (most cited first, 30 of 31)

- [Efficient Large Language Models: A Survey](https://mlsanity.com/paper/2312.03863/) (2023)
- [DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Models](https://mlsanity.com/paper/2402.19481/) (2024)
- [AsyncDiff: Parallelizing Diffusion Models by Asynchronous Denoising](https://mlsanity.com/paper/2406.06911/) (2024)
- [LLMServingSim 2.0: A Unified Simulator for Heterogeneous and Disaggregated LLM Serving Infrastructure](https://mlsanity.com/paper/2602.23036/) (2026)
- [Taming the Chaos: Coordinated Autoscaling for Heterogeneous and Disaggregated LLM Inference](https://mlsanity.com/paper/2508.19559/) (2025)
- [WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Serving](https://mlsanity.com/paper/2512.09472/) (2025)
- [PrefillShare: A Shared Prefill Module for KV Reuse in Multi-LLM Disaggregated Serving](https://mlsanity.com/paper/2602.12029/) (2026)
- [ShadowServe: Interference-Free KV Cache Fetching for Distributed Prefix Caching](https://mlsanity.com/paper/2509.16857/) (2025)
- [VoltanaLLM: Energy-Efficient and SLO-Aware Disaggregated LLM Serving via Adaptive Frequency Control and State-Space Routing](https://mlsanity.com/paper/2509.04827/) (2025)
- [Rethinking Latency Denial-of-Service: Attacking the LLM Serving Framework, Not the Model](https://mlsanity.com/paper/2602.07878/) (2026)
- [Token-Budget-Aware Pool Routing for Cost-Efficient LLM Inference](https://mlsanity.com/paper/2604.09613/) (2026)
- [Dual-Pool Token-Budget Routing for Cost-Efficient and Reliable LLM Serving](https://mlsanity.com/paper/2604.08075/) (2026)
- [CacheFlow: Efficient LLM Serving with 3D-Parallel KV Cache Restoration](https://mlsanity.com/paper/2604.25080/) (2026)
- [NetKV: Network-Aware Decode Instance Selection for Disaggregated LLM Inference](https://mlsanity.com/paper/2606.03910/) (2026)
- [DiLaServe: High SLO Attainment Serving for Diffusion Language Models](https://mlsanity.com/paper/2606.29094/) (2026)
- [Decomposing Runtime, Kernel, and Quantization Speedups via a Matched FP16 Intermediate: A Hardware-Conditioned Case Study on Four NVIDIA RTX A5000 GPUs](https://mlsanity.com/paper/2607.11368/) (2026)
- [Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving](https://mlsanity.com/paper/2608.06557/) (2026)
- [Queueing-Aware Optimization of Reasoning Tokens for Accuracy-Latency Trade-offs in LLM Servers](https://mlsanity.com/paper/2601.10274/) (2026)
- [Scalable Joint Resource Allocation for SLO-Constrained LLM Inference in Heterogeneous GPU Clouds](https://mlsanity.com/paper/2604.07472/) (2026)
- [SURGE: SuperBatch Unified Resource-efficient GPU Encoding for Heterogeneous Partitioned Data](https://mlsanity.com/paper/2605.01060/) (2026)
- [KV-RM: Regularizing KV-Cache Movement for Static-Graph LLM Serving](https://mlsanity.com/paper/2605.09735/) (2026)
- [Load Testing for Machine Learning Model Serving Systems at Scale](https://mlsanity.com/paper/2606.22013/) (2026)
- [From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving](https://mlsanity.com/paper/2607.02574/) (2026)
- [Metronome: Bound the Cache, Keep the Beat for Real-Time Interaction Model Serving](https://mlsanity.com/paper/2607.02640/) (2026)
- [JoyNexus: Service-Oriented Multi-Tenant Post-Training for VLA Models](https://mlsanity.com/paper/2607.16074/) (2026)
- [TrimMoE A communication aware and adaptive depth framework for distributed edge inference](https://mlsanity.com/paper/2608.00573/) (2026)
- [Beyond Binary Priorities: Multi-Tier SLA Scheduling for Large Language Model Serving](https://mlsanity.com/paper/2608.16336/) (2026)
- [FleetSieve: Decision-Critical Profiling for SLO-Aware LLM Fleet Configuration](https://mlsanity.com/paper/2608.19659/) (2026)
- [DART-FL: Burst-Aware Multitask Federated Learning under Dynamic Inference Demand at the Edge](https://mlsanity.com/paper/2608.27713/) (2026)
- [Calibrate, Then Route: A Measured Study of Learned Request Routing for Disaggregated LLM Serving](https://mlsanity.com/paper/2609.16206/) (2026)

## Abstract (arXiv)

Model parallelism is conventionally viewed as a method to scale a single large deep learning model beyond the memory limits of a single device. In this paper, we demonstrate that model parallelism can be additionally used for the statistical multiplexing of multiple devices when serving multiple models, even when a single model can fit into a single device. Our work reveals a fundamental trade-off between the overhead introduced by model parallelism and the opportunity to exploit statistical multiplexing to reduce serving latency in the presence of bursty workloads. We explore the new trade-off space and present a novel serving system, AlpaServe, that determines an efficient strategy for placing and parallelizing collections of large deep learning models across a distributed cluster. Evaluation results on production workloads show that AlpaServe can process requests at up to 10x higher rates or 6x more burstiness while staying within latency constraints for more than 99% of requests.