mlsanityData to 7 Oct 2026

P/D-Serve: Serving Disaggregated Large Language Model at Scale

arXiv 2408.08147 ↗First posted 15 Aug 2024cs.DC cs.CL cs.LGcited by 27

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Serving disaggregated large language models (LLMs) over tens of thousands of xPU devices (GPUs or NPUs) with reliable performance faces multiple challenges. 1) Ignoring the diversity (various prefixes and tidal requests), treating all the prompts in a mixed pool is inadequate. To facilitate the similarity per scenario and minimize the inner mismatch on P/D (prefill and decoding) processing, fine-grained organization is required, dynamically adjusting P/D ratios for better performance. 2) Due to inaccurate estimation on workload (queue status or maintained connections), the global scheduler easily incurs unnecessary timeouts in prefill. 3) Block-fixed device-to-device (D2D) KVCache transfer over cluster-level RDMA (remote direct memory access) fails to achieve desired D2D utilization as expected. To overcome previous problems, this paper proposes an end-to-end system P/D-Serve, complying with the paradigm of MLOps (machine learning operations), which models end-to-end (E2E) P/D performance and enables: 1) fine-grained P/D organization, mapping the service with RoCE (RDMA over converged ethernet) as needed, to facilitate similar processing and dynamic adjustments on P/D ratios; 2) on-demand forwarding upon rejections for idle prefill, decoupling the scheduler from regular inaccurate reports and local queues, to avoid timeouts in prefill; and 3) efficient KVCache transfer via optimized D2D access. P/D-Serve is implemented upon Ascend and MindSpore, has been deployed over tens of thousands of NPUs for more than eight months in commercial use, and further achieves 60%, 42% and 46% improvements on E2E throughput, time-to-first-token (TTFT) SLO (service level objective) and D2D transfer time. As the E2E system with optimizations, P/D-Serve achieves 6.7x increase on throughput, compared with aggregated LLMs.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 33 of its 45 references that are in the map. First posted on arXiv 15 Aug 2024.

Builds on

33 of 45 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool25 Jun
  2. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving24 Jun
  3. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism15 Apr
  4. 2024MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving2 Apr
  5. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  6. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  7. 2023Fairness in Serving Large Language Models31 Dec
  8. 2023FlashDecoding++: Faster Large Language Model Inference on GPUs2 Nov
  9. 2023Fast Distributed Inference Serving for Large Language Models10 May

Other references · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  6. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  7. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  8. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  9. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  10. 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 612
  11. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  12. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
12 more
  1. 2022Efficiently Scaling Transformer Inferencecited by 529
  2. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 450
  3. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  4. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 359
  5. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
  6. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
  7. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 159
  8. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 120
  9. 2024ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partitioncited by 81
  10. 2023PanGu-Sigma: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computingcited by 55
  11. 2023LlamaRec: Two-Stage Recommendation using Large Language Models for Rankingcited by 52
  12. 2024D'ej`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Servingcited by 48

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 14; 2026: 8.

27 papers.

Sort:

By area:

  1. 2024TurboRAG: Accelerating Retrieval-Augmented Generation with Precomputed KV Caches for Chunked Textcited by 60
  2. 2024BatchLLM: Optimizing Large Batched LLM Inference with Global Prefix Sharing and Throughput-oriented Token Batchingcited by 31
  3. 2025ServeGen: Workload Characterization and Generation of Large Language Model Serving in Productioncited by 29
  4. 2025ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environmentscited by 23
  5. 2024Efficiently Serving Large Multimodal Models Using EPD Disaggregationcited by 16
  6. 2025Taming the Titans: A Survey of Efficient LLM Inference Servingcited by 16
  7. 2025Seesaw: High-throughput LLM Inference via Model Re-shardingcited by 15
  8. 2025TokenFlow: Responsive LLM Text Streaming Serving under Request Burst via Preemptive Schedulingcited by 10
  9. 2025Beyond the Buzz: A Pragmatic Take on Inference Disaggregationcited by 9
  10. 2026LLMServingSim 2.0: A Unified Simulator for Heterogeneous and Disaggregated LLM Serving Infrastructurecited by 8
  11. 2025ElasticMM: Efficient Multimodal LLMs Serving with Elastic Multimodal Parallelismcited by 7
  12. 2024EPS-MoE: Expert Pipeline Scheduler for Cost-Efficient MoE Inferencecited by 6
15 more
  1. 2025Arrow: Adaptive Scheduling Mechanisms for Disaggregated LLM Inference Architecturecited by 5
  2. 2025Sparser Block-Sparse Attention via Token Permutationcited by 5
  3. 2026LAPS: A Length-Aware-Prefill LLM Serving Systemcited by 4
  4. 2025Disaggregated Prefill and Decoding Inference System for Large Language Model Serving on Multi-Vendor GPUscited by 3
  5. 2025ElasticMoE: An Efficient Auto Scaling Method for Mixture-of-Experts Modelscited by 3
  6. 2025xLLM Technical Reportcited by 3
  7. 2026RelayGR: Scaling Long-Sequence Generative Recommendation via Cross-Stage Relay-Race Inferencecited by 2
  8. 2024Tackling the Dynamicity in a Production LLM Serving System with SOTA Optimizations via Hybrid Prefill/Decode/Verify Scheduling on Efficient Meta-kernelscited by 1
  9. 2025P/D-Device: Disaggregated Large Language Model between Cloud and Devicescited by 1
  10. 2026SLO-Aware Compute Resource Allocation for Prefill-Decode Disaggregated LLM Inferencecited by 1
  11. 2025OmniInfer: System-Wide Acceleration Techniques for Optimizing LLM Serving Throughput and Latencycited by 0
  12. 2026Tropical: Enhancing SLO Attainment in Disaggregated LLM Serving via SLO-Aware Multiplexingcited by 0
  13. 20263DLS: A 3D Logic-Stacked Architecture for Disaggregated LLM Servingcited by 0
  14. 2026From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Servingcited by 0
  15. 2026FluidPD: In-Place Elasticity for SLO-Aware Prefill-Decode Disaggregated LLM Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025semi-PD: Towards Efficient LLM Serving via Phase-Wise Disaggregated Computation and Unified Storage
  2. 2025Revisiting Disaggregated Large Language Model Serving for Performance and Energy Implications
  3. 2024KVDirect: Distributed Disaggregated LLM Inference
  4. 2024NanoFlow: Towards Optimal Large Language Model Serving Throughput
  5. 2026RAPID-Serve: Resource-efficient and Accelerated P/D Intra-GPU Disaggregation
  6. 2026Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemption
  7. 2025DOPD: A Dynamic PD-Disaggregation Architecture for Maximizing Goodput in LLM Inference Serving
  8. 2025Mell: Memory-Efficient Large Language Model Serving via Multi-GPU KV Cache Management