mlsanityData to 14 Oct 2026

semi-PD: Towards Efficient LLM Serving via Phase-Wise Disaggregated Computation and Unified Storage

arXiv 2504.19867 ↗First posted 28 Apr 2025cs.CL cs.DC cs.LGcited by 16

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Existing large language model (LLM) serving systems fall into two categories: 1) a unified system where prefill phase and decode phase are co-located on the same GPU, sharing the unified computational resource and storage, and 2) a disaggregated system where the two phases are disaggregated to different GPUs. The design of the disaggregated system addresses the latency interference and sophisticated scheduling issues in the unified system but leads to storage challenges including 1) replicated weights for both phases that prevent flexible deployment, 2) KV cache transfer overhead between the two phases, 3) storage imbalance that causes substantial wasted space of the GPU capacity, and 4) suboptimal resource adjustment arising from the difficulties in migrating KV cache. Such storage inefficiency delivers poor serving performance under high request rates. In this paper, we identify that the advantage of the disaggregated system lies in the disaggregated computation, i.e., partitioning the computational resource to enable the asynchronous computation of two phases. Thus, we propose a novel LLM serving system, semi-PD, characterized by disaggregated computation and unified storage. In semi-PD, we introduce a computation resource controller to achieve disaggregated computation at the streaming multi-processor (SM) level, and a unified memory manager to manage the asynchronous memory access from both phases. semi-PD has a low-overhead resource adjustment mechanism between the two phases, and a service-level objective (SLO) aware dynamic partitioning algorithm to optimize the SLO attainment. Compared to state-of-the-art systems, semi-PD maintains lower latency at higher request rates, reducing the average end-to-end latency per request by 1.27-2.58x on DeepSeek series models, and serves 1.55-1.72x more requests adhering to latency constraints on Llama series models.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 34 of its 48 references that are in the map. First posted on arXiv 28 Apr 2025.

Builds on

34 of 48 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024Efficient LLM Scheduling by Learning to Rank28 Aug
  2. 2024NanoFlow: Towards Optimal Large Language Model Serving Throughput22 Aug
  3. 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool25 Jun
  4. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving24 Jun
  5. 2024A Survey on Efficient Inference for Large Language Models22 Apr
  6. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism15 Apr
  7. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  8. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  9. 2023Fairness in Serving Large Language Models31 Dec
  10. 2023FlashDecoding++: Faster Large Language Model Inference on GPUs2 Nov
  11. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills31 Aug
  12. 2023Fast Distributed Inference Serving for Large Language Models10 May

Other references · 22

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2024The Llama 3 Herd of Modelscited by 14,715
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  10. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  11. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  12. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
10 more
  1. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,310
  2. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  3. 2020Recipes for building an open-domain chatbotcited by 593
  4. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  5. 2020Towards a Human-like Open-Domain Chatbotcited by 470
  6. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  7. 2020IntelliCode Compose: Code Generation Using Transformercited by 227
  8. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 121
  9. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  10. 2024ExeGPT: Constraint-Aware Resource Scheduling for LLM Inferencecited by 45

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 9.

16 papers.

Sort:

By area:

  1. 2025ServeGen: Workload Characterization and Generation of Large Language Model Serving in Productioncited by 29
  2. 2025Arrow: Adaptive Scheduling Mechanisms for Disaggregated LLM Inference Architecturecited by 5
  3. 2025eLLM: Elastic Memory Management Framework for Efficient LLM Servingcited by 5
  4. 2025DuetServe: Harmonizing Prefill and Decode for LLM Serving via Adaptive GPU Multiplexingcited by 4
  5. 2025BanaServe: Unified KV Cache and Dynamic Module Migration for Balancing Disaggregated LLM Serving in AI Infrastructurecited by 3
  6. 2026FlowPrefill: Decoupling Preemption from Prefill Scheduling Granularity to Mitigate Head-of-Line Blocking in LLM Servingcited by 3
  7. 2026Prefilling-dLLM: Predictive Prefilling for Long-Context Inference in Diffusion Language Modelscited by 3
  8. 2025Revisiting Disaggregated Large Language Model Serving for Performance and Energy Implicationscited by 2
  9. 2025Enabling Disaggregated Multi-Stage MLLM Inference via GPU-Internal Scheduling and Resource Sharingcited by 1
  10. 2026Rollplex: Cross-Phase GPU Spatial Sharing for Vision Language Model Post-Trainingcited by 1
  11. 2026AInfer-PD: Communication-Safe In-Place Prefill-Decode Multiplexing for Distributed MoE Rolloutscited by 1
  12. 2026Threshold-Based Exclusive Batching for LLM Inferencecited by 0
4 more
  1. 2026FlexNPU: Transparent NPU Virtualization for Dynamic LLM Prefill-Decode Co-locationcited by 0
  2. 2026OpScale: Operator-level Provisioning and Autoscaling for LLM Servingcited by 0
  3. 2026EAServe: Encode-Aware Disaggregated Serving for Multimodal Large Language Modelscited by 0
  4. 2026Prorsa: Unleashing the Potential of Prefill-Decode Disaggregation in Agentic Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024KVDirect: Distributed Disaggregated LLM Inference
  2. 2024FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelines
  3. 2025Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs
  4. 2024ConServe: Fine-Grained GPU Harvesting for LLM Online and Offline Co-Serving
  5. 2026Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemption
  6. 2025Mell: Memory-Efficient Large Language Model Serving via Multi-GPU KV Cache Management
  7. 2026RAPID-Serve: Resource-efficient and Accelerated P/D Intra-GPU Disaggregation
  8. 2026SLIM: Saturation-Aware Lightweight Performance Modeling for LLM Serving