PDD: Unleashing Economical and Flexible Heterogeneous LLM Inference via Cross-Datacenter Prefill-Decode Disaggregation
On the map: Efficient LLM inference and serving → Serving systems and request scheduling
Abstract (arXiv)
Interconnecting geographically dispersed clusters over wide-area Ethernet provides a scalable and cost-effective alternative to dedicated intra-datacenter heterogeneous clusters for Large Language Model (LLM) inference. However, this cross-datacenter disaggregation imposes heavy KV-cache transfers between clusters, causing substantial Time-to-First-Token (TTFT) latency, which is especially detrimental for agentic workloads characterized by long contexts, high cache hit rates, and short outputs. We propose PDD, a three-tier disaggregation architecture built upon prefill-decode (PD) disaggregation, consisting of Prefill, RelayDecode (RLD), and MainDecode (MD) instances. On Cluster A, Prefill instances perform the prefill computation, while RLD instances immediately receive the KV cache via high-speed RDMA and begin decoding, thereby overlapping the KV transfer to Cluster B over TCP-based Ethernet. MD instances on Cluster B receive the KV cache along with the tokens produced by RLD, and decoding is then seamlessly handed off from RLD to MD for completion. To maximize overall efficiency, PDD employs three core mechanisms: Decode-side RadixCache to alleviate bandwidth bottlenecks, an Extend-Decode Handoff mechanism for smooth control migration between RLD and MD, and multi-stage pipeline orchestration to manage complex inter-tier dependencies under high concurrency and long-term serving. We further design a low-cost, fine-grained heterogeneous deployment scheme that maximizes latency-masking efficiency at marginal cost. Compared to the intra-DC homogeneous PD baseline, PDD's cross-datacenter mapping of compute-intensive H100s and memory-bandwidth-optimized H200s achieves a Benefit-Cost Ratio (BCR) up to 37.5% higher in SLA-compliant goodput.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 8 of its 10 references that are in the map. First posted on arXiv 18 Jul 2026.
Builds on
8 of 10 references.
Closest ancestors · 2
Same direction, newest first
- 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving24 Jun
- 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
Other references · 6
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
- 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
- 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
- 2026Prefill-as-a-Service: KVCache of Next-Generation Models Could Go Cross-Datacentercited by 11
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026SmartGen: Seamless Disaggregated LLM Inference with Selective KV Cache Transfer
- 2025DOPD: A Dynamic PD-Disaggregation Architecture for Maximizing Goodput in LLM Inference Serving
- 2026SLO-Aware Compute Resource Allocation for Prefill-Decode Disaggregated LLM Inference
- 2024KVDirect: Distributed Disaggregated LLM Inference
- 2026Beyond Prefill-Decode Disaggregation: Dissecting LLM Inference for Heterogeneous Platforms via Dynamic Operator Scheduling
- 2025HACK: Homomorphic Acceleration via Compression of the Key-Value Cache for Disaggregated LLM Inference
- 2025Arrow: Adaptive Scheduling Mechanisms for Disaggregated LLM Inference Architecture
- 2025HexGen-2: Disaggregated Generative Inference of LLMs in Heterogeneous Environment