Disaggregated Prefill and Decoding Inference System for Large Language Model Serving on Multi-Vendor GPUs
On the map: Efficient LLM inference and serving → Serving systems and request scheduling
Abstract (arXiv)
LLM-based applications have been widely used in various industries, but with the increasing of models size, an efficient large language model (LLM) inference system is an urgent problem to be solved for service providers. Since the inference system is divided into two stage with different characteristics: Prefill and Decode, the two stage will interfere with each other during the inference process. Toward this end, a P-D disaggregated inference framework is proposed by some researchers. Current research is done on homogeneous GPUs, and lacks deployment solutions based on business scenarios. Compared with homogeneous GPUs, using heterogeneous GPUs to construct inference systems can better improve resource utilization and reduce costs. Even if GPUs from different vendors are used to build inference systems, on the basis of reducing costs, the resource utilization rate can be improved and the dependence on a single vendor can be reduced. Therefore, a P-D disaggreagetd inference system based on heterogeneous GPUs is designed, and the heterogeneous compatible transmission module in the system is designed to address heterogeneous GPU data compatibility issues. Then, a joint optimization algorithm of parallel strategy and instance number allocation is proposed to obtain the deployment solutions. Finally, the experimental results show that the P-D disaggregated inference system can well solve the hybrid inference problem of heterogeneous GPUs from different vendors, and the joint optimization algorithm can obtain the optimal deployment solution.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 15 of its 21 references that are in the map. First posted on arXiv 22 Sep 2025.
Builds on
15 of 21 references.
Closest ancestors · 7
Same direction, newest first
- 2025Arrow: Adaptive Scheduling Mechanisms for Disaggregated LLM Inference Architecture17 May
- 2025FlowKV: A Disaggregated Inference Framework with Low-Latency KV Cache Transfer and Load-Aware Scheduling3 Apr
- 2025Injecting Adrenaline into LLM Serving: Boosting Resource Utilization and Throughput via Attention Disaggregation26 Mar
- 2024KVDirect: Distributed Disaggregated LLM Inference13 Dec
- 2024P/D-Serve: Serving Disaggregated Large Language Model at Scale15 Aug
- 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving24 Jun
- 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills31 Aug
Other references · 8
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
- 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
- 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 121
- 2024LLM Inference Serving: Survey of Recent Advances and Opportunitiescited by 46
- 2024Understanding User Experience in Large Language Model Interactionscited by 44
- 2025HexGen-2: Disaggregated Generative Inference of LLMs in Heterogeneous Environmentcited by 20
- 2024Canvil: Designerly Adaptation for LLM-Powered User Experiencescited by 9
Built on by
Citing papers per year: 2026: 3.
3 papers.
- 2026HBM Is Not All You Need: Efficient Disaggregated LLM Serving across Memory-heterogeneous Accelerators29 Jun 2026
- 2026SLO-Aware Compute Resource Allocation for Prefill-Decode Disaggregated LLM Inference5 Mar 2026
- 2026TIDE: Temporal Incremental Draft Engine for Self-Improving LLM Inference5 Feb 2026
Similar, not cited
Close in wording, no citation either way
- 2025SPAD: Specialized Prefill and Decode Hardware for Disaggregated LLM Inference
- 2025semi-PD: Towards Efficient LLM Serving via Phase-Wise Disaggregated Computation and Unified Storage
- 2024Dovetail: A CPU/GPU Heterogeneous Speculative Decoding for LLM inference
- 2026When Does Disaggregation Pay? Simulating Prefill--Decode--Attention--FFN Specialization for Agentic LLM Inference
- 2023Efficient LLM inference solution on Intel GPU
- 2024LLM-PQ: Serving LLM on Heterogeneous Clusters with Phase-Aware Partition and Adaptive Quantization
- 2026Unified Static-Dynamic Pruning for Efficient LLM Inference
- 2023The Synergy of Speculative Decoding and Batching in Serving Large Language Models