# MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving

arXiv: https://arxiv.org/abs/2404.02015 · HTML: https://mlsanity.com/paper/2404.02015/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving.
- Builds on 15 of its 32 references that are in the map.
- First posted on arXiv 2 Apr 2024.

## Placed in

- Efficient LLM inference and serving / [Serving systems and request scheduling](https://mlsanity.com/inference/serving-systems-and-request-scheduling/)

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU](https://mlsanity.com/paper/2303.06865/) (2023)
- [DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale](https://mlsanity.com/paper/2207.00032/) (2022)
- [Clipper: A Low-Latency Online Prediction Serving System](https://mlsanity.com/paper/1612.03079/) (2016)
- [Serving DNNs like Clockwork: Performance Predictability from the Bottom Up](https://mlsanity.com/paper/2006.02464/) (2020)
- [SpotServe: Serving Generative Large Language Models on Preemptible Instances](https://mlsanity.com/paper/2311.15566/) (2023)
- [TensorFlow-Serving: Flexible, High-Performance ML Serving](https://mlsanity.com/paper/1712.06139/) (2017)
- [Fairness in Serving Large Language Models](https://mlsanity.com/paper/2401.00588/) (2023)
- [Serving DNN Models with Multi-Instance GPUs: A Case of the Reconfigurable Machine Scheduling Problem](https://mlsanity.com/paper/2109.11067/) (2021)
- [MuxFlow: Efficient and Safe GPU Sharing in Large-Scale Production Deep Learning Clusters](https://mlsanity.com/paper/2303.13803/) (2023)

## Built on by (most cited first, 30 of 52)

- [OmniRouter: Budget and Performance Controllable Multi-LLM Routing](https://mlsanity.com/paper/2502.20576/) (2025)
- [ServeGen: Workload Characterization and Generation of Large Language Model Serving in Production](https://mlsanity.com/paper/2505.09999/) (2025)
- [P/D-Serve: Serving Disaggregated Large Language Model at Scale](https://mlsanity.com/paper/2408.08147/) (2024)
- [Parallelizing Tool Execution and LLM Generation for Low-Latency Agent Serving](https://mlsanity.com/paper/2603.18897/) (2026)
- [BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batching](https://mlsanity.com/paper/2411.16102/) (2024)
- [Taming the Titans: A Survey of Efficient LLM Inference Serving](https://mlsanity.com/paper/2504.19720/) (2025)
- [HydraServe: Minimizing Cold Start Latency for Serverless LLM Serving in Public Clouds](https://mlsanity.com/paper/2502.15524/) (2025)
- [Nexus:Proactive Intra-GPU Disaggregation of Prefill and Decode in LLM Serving](https://mlsanity.com/paper/2507.06608/) (2025)
- [TokenScale: Timely and Accurate Autoscaling for Disaggregated LLM Serving with Token Velocity](https://mlsanity.com/paper/2512.03416/) (2025)
- [HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-location](https://mlsanity.com/paper/2501.14808/) (2025)
- [FlexPipe: Adapting Dynamic LLM Serving Through Inflight Pipeline Refactoring in Fragmented Serverless Clusters](https://mlsanity.com/paper/2510.11938/) (2025)
- [KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Serving](https://mlsanity.com/paper/2605.13734/) (2026)
- [Pythia: Exploiting Workflow Predictability for Efficient Agent-Native LLM Serving](https://mlsanity.com/paper/2604.25899/) (2026)
- [Cascadia: An Efficient Cascade Serving System for Large Language Models](https://mlsanity.com/paper/2506.04203/) (2025)
- [WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Serving](https://mlsanity.com/paper/2512.09472/) (2025)
- [Boosting LLM Serving through Spatial-Temporal GPU Resource Sharing](https://mlsanity.com/paper/2504.19516/) (2025)
- [TetriServe: Efficiently Serving Mixed DiT Workloads](https://mlsanity.com/paper/2510.01565/) (2025)
- [DuetServe: Harmonizing Prefill and Decode for LLM Serving via Adaptive GPU Multiplexing](https://mlsanity.com/paper/2511.04791/) (2025)
- [PrefillShare: A Shared Prefill Module for KV Reuse in Multi-LLM Disaggregated Serving](https://mlsanity.com/paper/2602.12029/) (2026)
- [FineServe: Precision-Aware KV Slab and Two-Level Scheduling for Heterogeneous Precision LLM Serving](https://mlsanity.com/paper/2509.06261/) (2025)
- [Potential Applications of HBF in LLM Serving Systems](https://mlsanity.com/paper/2608.13127/) (2026)
- [CASTILLO: Characterizing Response Length Distributions of Large Language Models](https://mlsanity.com/paper/2505.16881/) (2025)
- [MoE-Prism: Disentangling Monolithic Experts for Elastic MoE Services via Model-System Co-Designs](https://mlsanity.com/paper/2510.19366/) (2025)
- [TIDE: Temporal Incremental Draft Engine for Self-Improving LLM Inference](https://mlsanity.com/paper/2602.05145/) (2026)
- [Scepsy: Serving Agentic Workflows Using Aggregate LLM Pipelines](https://mlsanity.com/paper/2604.15186/) (2026)
- [JigsawRL: Assembling RL Pipelines for Efficient LLM Post-Training](https://mlsanity.com/paper/2604.23838/) (2026)
- [CrossPool: Efficient Multi-LLM Serving for Cold MoE Models through KV-Cache and Weight Disaggregation](https://mlsanity.com/paper/2606.24506/) (2026)
- [TRACE-ROUTER: Task-Consistent and Adaptive Online Routing for Agentic AI](https://mlsanity.com/paper/2607.22465/) (2026)
- [Reducing the Cost of Dropout in Flash-Attention by Hiding RNG with GEMM](https://mlsanity.com/paper/2410.07531/) (2024)
- [Improving the End-to-End Efficiency of Offline Inference for Multi-LLM Applications Based on Sampling and Simulation](https://mlsanity.com/paper/2503.16893/) (2025)

## Abstract (arXiv)

Large language models (LLMs) have demonstrated remarkable performance, and organizations are racing to serve LLMs of varying sizes as endpoints for use-cases like chat, programming and search. However, efficiently serving multiple LLMs poses significant challenges for existing approaches due to varying popularity of LLMs. In the paper, we present MuxServe, a flexible spatial-temporal multiplexing system for efficient multiple LLM serving. The key insight behind is to colocate LLMs considering their popularity to multiplex memory resources, and leverage the characteristics of prefill and decoding phases to separate and flexibly colocate them to multiplex computation resources. MuxServe formally formulates the multiplexing problem, and proposes a novel placement algorithm and adaptive batch scheduling strategy to identify optimal colocations and maximize utilization. MuxServe designs a unified resource manager to enable flexible and efficient multiplexing. Evaluation results show that MuxServe can achieves up to 1.8× higher throughput or processes 2.9× more requests within 99% SLO attainment. The code is available at: urlhttps://github.com/hao-ai-lab/MuxServe.