# Disaggregated Prefill and Decoding Inference System for Large Language Model Serving on Multi-Vendor GPUs

arXiv: https://arxiv.org/abs/2509.17542 · HTML: https://mlsanity.com/paper/2509.17542/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving.
- Builds on 15 of its 21 references that are in the map.
- First posted on arXiv 22 Sep 2025.

## Placed in

- Efficient LLM inference and serving / [Serving systems and request scheduling](https://mlsanity.com/inference/serving-systems-and-request-scheduling/)

## Builds on (most cited first, 15 of 15)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM](https://mlsanity.com/paper/2104.04473/) (2021)
- [Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving](https://mlsanity.com/paper/2407.00079/) (2024)
- [SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills](https://mlsanity.com/paper/2308.16369/) (2023)
- [Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads](https://mlsanity.com/paper/2401.11181/) (2024)
- [LLM Inference Serving: Survey of Recent Advances and Opportunities](https://mlsanity.com/paper/2407.12391/) (2024)
- [Understanding User Experience in Large Language Model Interactions](https://mlsanity.com/paper/2401.08329/) (2024)
- [P/D-Serve: Serving Disaggregated Large Language Model at Scale](https://mlsanity.com/paper/2408.08147/) (2024)
- [HexGen-2: Disaggregated Generative Inference of LLMs in Heterogeneous Environment](https://mlsanity.com/paper/2502.07903/) (2025)
- [Injecting Adrenaline into LLM Serving: Boosting Resource Utilization and Throughput via Attention Disaggregation](https://mlsanity.com/paper/2503.20552/) (2025)
- [FlowKV: A Disaggregated Inference Framework with Low-Latency KV Cache Transfer and Load-Aware Scheduling](https://mlsanity.com/paper/2504.03775/) (2025)
- [Canvil: Designerly Adaptation for LLM-Powered User Experiences](https://mlsanity.com/paper/2401.09051/) (2024)
- [KVDirect: Distributed Disaggregated LLM Inference](https://mlsanity.com/paper/2501.14743/) (2024)
- [Arrow: Adaptive Scheduling Mechanisms for Disaggregated LLM Inference Architecture](https://mlsanity.com/paper/2505.11916/) (2025)

## Built on by (most cited first, 3 of 3)

- [TIDE: Temporal Incremental Draft Engine for Self-Improving LLM Inference](https://mlsanity.com/paper/2602.05145/) (2026)
- [SLO-Aware Compute Resource Allocation for Prefill-Decode Disaggregated LLM Inference](https://mlsanity.com/paper/2603.04716/) (2026)
- [HBM Is Not All You Need: Efficient Disaggregated LLM Serving across Memory-heterogeneous Accelerators](https://mlsanity.com/paper/2606.29986/) (2026)

## Abstract (arXiv)

LLM-based applications have been widely used in various industries, but with the increasing of models size, an efficient large language model (LLM) inference system is an urgent problem to be solved for service providers. Since the inference system is divided into two stage with different characteristics: Prefill and Decode, the two stage will interfere with each other during the inference process. Toward this end, a P-D disaggregated inference framework is proposed by some researchers. Current research is done on homogeneous GPUs, and lacks deployment solutions based on business scenarios. Compared with homogeneous GPUs, using heterogeneous GPUs to construct inference systems can better improve resource utilization and reduce costs. Even if GPUs from different vendors are used to build inference systems, on the basis of reducing costs, the resource utilization rate can be improved and the dependence on a single vendor can be reduced. Therefore, a P-D disaggreagetd inference system based on heterogeneous GPUs is designed, and the heterogeneous compatible transmission module in the system is designed to address heterogeneous GPU data compatibility issues. Then, a joint optimization algorithm of parallel strategy and instance number allocation is proposed to obtain the deployment solutions. Finally, the experimental results show that the P-D disaggregated inference system can well solve the hybrid inference problem of heterogeneous GPUs from different vendors, and the joint optimization algorithm can obtain the optimal deployment solution.