mlsanityData to 7 Oct 2026

AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving

arXiv 2302.11665 ↗First posted 22 Feb 2023cs.LG cs.DC cs.NIcited by 31

Not on the map. Why?

Abstract (arXiv)

Model parallelism is conventionally viewed as a method to scale a single large deep learning model beyond the memory limits of a single device. In this paper, we demonstrate that model parallelism can be additionally used for the statistical multiplexing of multiple devices when serving multiple models, even when a single model can fit into a single device. Our work reveals a fundamental trade-off between the overhead introduced by model parallelism and the opportunity to exploit statistical multiplexing to reduce serving latency in the presence of bursty workloads. We explore the new trade-off space and present a novel serving system, AlpaServe, that determines an efficient strategy for placing and parallelizing collections of large deep learning models across a distributed cluster. Evaluation results on production workloads show that AlpaServe can process requests at up to 10x higher rates or 6x more burstiness while staying within latency constraints for more than 99% of requests.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 10 of its 57 references that are in the map. First posted on arXiv 22 Feb 2023.

Builds on

10 of 57 references.

References · 10

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  6. 2022Efficiently Scaling Transformer Inferencecited by 529
  7. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  8. 2021GSPMD: General and Scalable Parallelization for ML Computation Graphscited by 109
  9. 2017TensorFlow-Serving: Flexible, High-Performance ML Servingcited by 85
  10. 2021Robustness Challenges in Model Distillation and Pruning for Natural Language Understandingcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 2; 2025: 4; 2026: 24.

31 papers.

Sort:

By area:

  1. 2023Efficient Large Language Models: A Surveycited by 183
  2. 2024DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Modelscited by 98
  3. 2024AsyncDiff: Parallelizing Diffusion Models by Asynchronous Denoisingcited by 22
  4. 2026LLMServingSim 2.0: A Unified Simulator for Heterogeneous and Disaggregated LLM Serving Infrastructurecited by 8
  5. 2025Taming the Chaos: Coordinated Autoscaling for Heterogeneous and Disaggregated LLM Inferencecited by 5
  6. 2025ShadowServe: Interference-Free KV Cache Fetching for Distributed Prefix Cachingcited by 3
  7. 2026PrefillShare: A Shared Prefill Module for KV Reuse in Multi-LLM Disaggregated Servingcited by 3
  8. 2025VoltanaLLM: Energy-Efficient and SLO-Aware Disaggregated LLM Serving via Adaptive Frequency Control and State-Space Routingcited by 2
  9. 2025WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Servingcited by 2
  10. 2026Rethinking Latency Denial-of-Service: Attacking the LLM Serving Framework, Not the Modelcited by 2
  11. 2026Token-Budget-Aware Pool Routing for Cost-Efficient LLM Inferencecited by 2
  12. 2026Dual-Pool Token-Budget Routing for Cost-Efficient and Reliable LLM Servingcited by 1
19 more
  1. 2026CacheFlow: Efficient LLM Serving with 3D-Parallel KV Cache Restorationcited by 1
  2. 2026NetKV: Network-Aware Decode Instance Selection for Disaggregated LLM Inferencecited by 1
  3. 2026DiLaServe: High SLO Attainment Serving for Diffusion Language Modelscited by 1
  4. 2026Decomposing Runtime, Kernel, and Quantization Speedups via a Matched FP16 Intermediate: A Hardware-Conditioned Case Study on Four NVIDIA RTX A5000 GPUscited by 1
  5. 2026Queueing-Aware Optimization of Reasoning Tokens for Accuracy-Latency Trade-offs in LLM Serverscited by 0
  6. 2026Scalable Joint Resource Allocation for SLO-Constrained LLM Inference in Heterogeneous GPU Cloudscited by 0
  7. 2026SURGE: SuperBatch Unified Resource-efficient GPU Encoding for Heterogeneous Partitioned Datacited by 0
  8. 2026KV-RM: Regularizing KV-Cache Movement for Static-Graph LLM Servingcited by 0
  9. 2026Load Testing for Machine Learning Model Serving Systems at Scalecited by 0
  10. 2026From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Servingcited by 0
  11. 2026Metronome: Bound the Cache, Keep the Beat for Real-Time Interaction Model Servingcited by 0
  12. 2026JoyNexus: Service-Oriented Multi-Tenant Post-Training for VLA Modelscited by 0
  13. 2026TrimMoE A communication aware and adaptive depth framework for distributed edge inferencecited by 0
  14. 2026Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference servingcited by 0
  15. 2026Beyond Binary Priorities: Multi-Tier SLA Scheduling for Large Language Model Servingcited by 0
  16. 2026FleetSieve: Decision-Critical Profiling for SLO-Aware LLM Fleet Configurationcited by 0
  17. 2026DART-FL: Burst-Aware Multitask Federated Learning under Dynamic Inference Demand at the Edgecited by 0
  18. 2026Calibrate, Then Route: A Measured Study of Learned Request Routing for Disaggregated LLM Servingcited by 0
  19. 2026WeightBridge: An Efficient Weight Transfer Library for Reinforcement Learningcited by 0