mlsanityData to 7 Oct 2026

MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving

arXiv 2404.02015 ↗First posted 2 Apr 2024cs.DCcited by 49

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Large language models (LLMs) have demonstrated remarkable performance, and organizations are racing to serve LLMs of varying sizes as endpoints for use-cases like chat, programming and search. However, efficiently serving multiple LLMs poses significant challenges for existing approaches due to varying popularity of LLMs. In the paper, we present MuxServe, a flexible spatial-temporal multiplexing system for efficient multiple LLM serving. The key insight behind is to colocate LLMs considering their popularity to multiplex memory resources, and leverage the characteristics of prefill and decoding phases to separate and flexibly colocate them to multiplex computation resources. MuxServe formally formulates the multiplexing problem, and proposes a novel placement algorithm and adaptive batch scheduling strategy to identify optimal colocations and maximize utilization. MuxServe designs a unified resource manager to enable flexible and efficient multiplexing. Evaluation results show that MuxServe can achieves up to 1.8× higher throughput or processes 2.9× more requests within 99% SLO attainment. The code is available at: urlhttps://github.com/hao-ai-lab/MuxServe.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 15 of its 32 references that are in the map. First posted on arXiv 2 Apr 2024.

Builds on

15 of 32 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Fairness in Serving Large Language Models31 Dec
  2. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instances27 Nov

Other references · 13

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  7. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  8. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  9. 2016Clipper: A Low-Latency Online Prediction Serving Systemcited by 191
  10. 2020Serving DNNs like Clockwork: Performance Predictability from the Bottom Upcited by 119
  11. 2017TensorFlow-Serving: Flexible, High-Performance ML Servingcited by 85
  12. 2021Serving DNN Models with Multi-Instance GPUs: A Case of the Reconfigurable Machine Scheduling Problemcited by 7
1 more
  1. 2023MuxFlow: Efficient and Safe GPU Sharing in Large-Scale Production Deep Learning Clusterscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 18; 2026: 28.

49 papers.

Sort:

By area:

  1. 2025OmniRouter: Budget and Performance Controllable Multi-LLM Routingcited by 35
  2. 2025ServeGen: Workload Characterization and Generation of Large Language Model Serving in Productioncited by 29
  3. 2024P/D-Serve: Serving Disaggregated Large Language Model at Scalecited by 27
  4. 2024BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batchingcited by 19
  5. 2025Taming the Titans: A Survey of Efficient LLM Inference Servingcited by 16
  6. 2026Parallelizing Tool Execution and LLM Generation for Low-Latency Agent Servingcited by 16
  7. 2025HydraServe: Minimizing Cold Start Latency for Serverless LLM Serving in Public Cloudscited by 12
  8. 2025Nexus:Proactive Intra-GPU Disaggregation of Prefill and Decode in LLM Servingcited by 12
  9. 2025HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-locationcited by 10
  10. 2025FlexPipe: Adapting Dynamic LLM Serving Through Inflight Pipeline Refactoring in Fragmented Serverless Clusterscited by 9
  11. 2025TokenScale: Timely and Accurate Autoscaling for Disaggregated LLM Serving with Token Velocitycited by 9
  12. 2026KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Servingcited by 7
37 more
  1. 2026Pythia: Exploiting Workflow Predictability for Efficient Agent-Native LLM Servingcited by 6
  2. 2025Cascadia: An Efficient Cascade Serving System for Large Language Modelscited by 5
  3. 2025Boosting LLM Serving through Spatial-Temporal GPU Resource Sharingcited by 4
  4. 2025TetriServe: Efficiently Serving Mixed DiT Workloadscited by 4
  5. 2025FineServe: Precision-Aware KV Slab and Two-Level Scheduling for Heterogeneous Precision LLM Servingcited by 3
  6. 2025DuetServe: Harmonizing Prefill and Decode for LLM Serving via Adaptive GPU Multiplexingcited by 3
  7. 2026PrefillShare: A Shared Prefill Module for KV Reuse in Multi-LLM Disaggregated Servingcited by 3
  8. 2025WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Servingcited by 2
  9. 2026Potential Applications of HBF in LLM Serving Systemscited by 2
  10. 2025CASTILLO: Characterizing Response Length Distributions of Large Language Modelscited by 1
  11. 2025MoE-Prism: Disentangling Monolithic Experts for Elastic MoE Services via Model-System Co-Designscited by 1
  12. 2026TIDE: Temporal Incremental Draft Engine for Self-Improving LLM Inferencecited by 1
  13. 2026Scepsy: Serving Agentic Workflows Using Aggregate LLM Pipelinescited by 1
  14. 2026CrossPool: Efficient Multi-LLM Serving for Cold MoE Models through KV-Cache and Weight Disaggregationcited by 1
  15. 2026TRACE-ROUTER: Task-Consistent and Adaptive Online Routing for Agentic AIcited by 1
  16. 2024Reducing the Cost of Dropout in Flash-Attention by Hiding RNG with GEMMcited by 0
  17. 2025Improving the End-to-End Efficiency of Offline Inference for Multi-LLM Applications Based on Sampling and Simulationcited by 0
  18. 2025From Models to Operators: Rethinking Autoscaling Granularity for Large Generative Modelscited by 0
  19. 2026SageSched: Efficient LLM Scheduling Confronting Demand Uncertainty and Hybriditycited by 0
  20. 2026LegoDiffusion: Micro-Serving Text-to-Image Diffusion Workflowscited by 0
  21. 2026RouterWise: Joint Resource Allocation and Routing for Latency-Aware Multi-Model LLM Servingcited by 0
  22. 2026Scalable AI Inference: Performance Analysis and Optimization of AI Model Servingcited by 0
  23. 2026JigsawRL: Assembling RL Pipelines for Efficient LLM Post-Trainingcited by 0
  24. 2026SPECTRE: Hybrid Ordinary-Parallel Speculative Serving for Resource-Efficient LLM Inferencecited by 0
  25. 2026Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemptioncited by 0
  26. 2026MOSAIC: Efficient Mixture-of-Agent Scheduling via Adaptive Aggregation and Inference Concurrencycited by 0
  27. 2026FlexNPU: Transparent NPU Virtualization for Dynamic LLM Prefill-Decode Co-locationcited by 0
  28. 2026Tropical: Enhancing SLO Attainment in Disaggregated LLM Serving via SLO-Aware Multiplexingcited by 0
  29. 2026Beyond Prediction: Tail-Aware Scheduling for LLM Inferencecited by 0
  30. 2026Toward Open Weight Models Without Risks: Separating Public and Private Capabilities in LLMscited by 0
  31. 2026Energy-Aware Scheduling for Serverless LLM Serving on Shared GPUscited by 0
  32. 2026From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Servingcited by 0
  33. 2026Elastic Gang: Per-Token Membership Change for a Hard-Barriered LLM Inference Gang Co-Scheduled with OS Processescited by 0
  34. 2026ServerlessT2I: Efficient Text-to-Image Workflow Serving on a Serverless Platformcited by 0
  35. 2026OpScale: Operator-level Provisioning and Autoscaling for LLM Servingcited by 0
  36. 2026Efficient Vision-Language-Action Management and Serving for Robot Factoriescited by 0
  37. 2026Cross-Model Autoscaling for Shared LLM Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026OServe: Accelerating LLM Serving via Spatial-Temporal Workload Orchestration
  2. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache
  3. 2024ScaleLLM: A Resource-Frugal LLM Serving Framework by Optimizing End-to-End Efficiency
  4. 2023MUX-PLMs: Data Multiplexing for High-throughput Language Models
  5. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving
  6. 2025Unlock the Potential of Fine-grained LLM Serving via Dynamic Module Scaling
  7. 2025AdaServe: Accelerating Multi-SLO LLM Serving with SLO-Customized Speculative Decoding
  8. 2025Optimal Scheduling Algorithms for LLM Inference: Theory and Practice