# Beyond the Buzz: A Pragmatic Take on Inference Disaggregation

arXiv: https://arxiv.org/abs/2506.05508 · HTML: https://mlsanity.com/paper/2506.05508/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 23 references that are in the map.
- First posted on arXiv 5 Jun 2025.

## Builds on (most cited first, 16 of 16)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving](https://mlsanity.com/paper/2401.09670/) (2024)
- [Splitwise: Efficient generative LLM inference using phase splitting](https://mlsanity.com/paper/2311.18677/) (2023)
- [Efficiently Scaling Transformer Inference](https://mlsanity.com/paper/2211.05102/) (2022)
- [DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale](https://mlsanity.com/paper/2207.00032/) (2022)
- [SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills](https://mlsanity.com/paper/2308.16369/) (2023)
- [Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads](https://mlsanity.com/paper/2401.11181/) (2024)
- [MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool](https://mlsanity.com/paper/2406.17565/) (2024)
- [D'ej`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Serving](https://mlsanity.com/paper/2403.01876/) (2024)
- [FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelines](https://mlsanity.com/paper/2403.11421/) (2024)
- [P/D-Serve: Serving Disaggregated Large Language Model at Scale](https://mlsanity.com/paper/2408.08147/) (2024)
- [HexGen-2: Disaggregated Generative Inference of LLMs in Heterogeneous Environment](https://mlsanity.com/paper/2502.07903/) (2025)
- [Efficient Heterogeneous Large Language Model Decoding with Model-Attention Disaggregation](https://mlsanity.com/paper/2405.01814/) (2024)
- [KVDirect: Distributed Disaggregated LLM Inference](https://mlsanity.com/paper/2501.14743/) (2024)

## Built on by (most cited first, 9 of 9)

- [LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Experts](https://mlsanity.com/paper/2601.18089/) (2026)
- [Efficient and Scalable Agentic AI with Heterogeneous Systems](https://mlsanity.com/paper/2507.19635/) (2025)
- [DuetServe: Harmonizing Prefill and Decode for LLM Serving via Adaptive GPU Multiplexing](https://mlsanity.com/paper/2511.04791/) (2025)
- [Revati: Transparent GPU-Free Time-Warp Emulation for LLM Serving](https://mlsanity.com/paper/2601.00397/) (2026)
- [From Tokens to Layers: Redefining Stall-Free Scheduling for MoE Serving with Layered Prefill](https://mlsanity.com/paper/2510.08055/) (2025)
- [Frontier: Simulating the Next Generation of LLM Inference Systems](https://mlsanity.com/paper/2508.03148/) (2025)
- [Sangam: Efficiently Serving Diffusion LLMs with the AR Stack](https://mlsanity.com/paper/2607.04206/) (2026)
- [RAPID-Serve: Resource-efficient and Accelerated P/D Intra-GPU Disaggregation](https://mlsanity.com/paper/2601.11822/) (2026)
- [FluidPD: In-Place Elasticity for SLO-Aware Prefill-Decode Disaggregated LLM Serving](https://mlsanity.com/paper/2610.06917/) (2026)

## Abstract (arXiv)

As inference scales to multi-node deployments, disaggregation - splitting inference into distinct phases - offers a promising path to improving the throughput-interactivity Pareto frontier. Despite growing enthusiasm and a surge of open-source efforts, practical deployment of disaggregated serving remains limited due to the complexity of the optimization search space and system-level coordination. In this paper, we present the first systematic study of disaggregated inference at scale, evaluating hundreds of thousands of design points across diverse workloads and hardware configurations. We find that disaggregation is most effective for prefill-heavy traffic patterns and larger models. Our results highlight the critical role of dynamic rate matching and elastic scaling in achieving Pareto-optimal performance. Our findings offer actionable insights for efficient disaggregated deployments to navigate the trade-off between system throughput and interactivity.