mlsanityData to 14 Oct 2026

Beyond the Buzz: A Pragmatic Take on Inference Disaggregation

arXiv 2506.05508 ↗First posted 5 Jun 2025cs.DC cs.AIcited by 9

Not on the map. Why?

Abstract (arXiv)

As inference scales to multi-node deployments, disaggregation - splitting inference into distinct phases - offers a promising path to improving the throughput-interactivity Pareto frontier. Despite growing enthusiasm and a surge of open-source efforts, practical deployment of disaggregated serving remains limited due to the complexity of the optimization search space and system-level coordination. In this paper, we present the first systematic study of disaggregated inference at scale, evaluating hundreds of thousands of design points across diverse workloads and hardware configurations. We find that disaggregation is most effective for prefill-heavy traffic patterns and larger models. Our results highlight the critical role of dynamic rate matching and elastic scaling in achieving Pareto-optimal performance. Our findings offer actionable insights for efficient disaggregated deployments to navigate the trade-off between system throughput and interactivity.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 16 of its 23 references that are in the map. First posted on arXiv 5 Jun 2025.

Builds on

16 of 23 references.

References · 16

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  3. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  4. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  5. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  6. 2022Efficiently Scaling Transformer Inferencecited by 539
  7. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  8. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 193
  9. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 121
  10. 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Poolcited by 68
  11. 2024FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelinescited by 49
  12. 2024D'ej`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Servingcited by 48
4 more
  1. 2024P/D-Serve: Serving Disaggregated Large Language Model at Scalecited by 27
  2. 2025HexGen-2: Disaggregated Generative Inference of LLMs in Heterogeneous Environmentcited by 20
  3. 2024Efficient Heterogeneous Large Language Model Decoding with Model-Attention Disaggregationcited by 12
  4. 2024KVDirect: Distributed Disaggregated LLM Inferencecited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 5.

9 papers.

  1. 2026LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Expertscited by 14
  2. 2025Efficient and Scalable Agentic AI with Heterogeneous Systemscited by 11
  3. 2026Revati: Transparent GPU-Free Time-Warp Emulation for LLM Servingcited by 5
  4. 2025From Tokens to Layers: Redefining Stall-Free Scheduling for MoE Serving with Layered Prefillcited by 4
  5. 2025DuetServe: Harmonizing Prefill and Decode for LLM Serving via Adaptive GPU Multiplexingcited by 4
  6. 2025Frontier: Simulating the Next Generation of LLM Inference Systemscited by 2
  7. 2026Sangam: Efficiently Serving Diffusion LLMs with the AR Stackcited by 2
  8. 2026RAPID-Serve: Resource-efficient and Accelerated P/D Intra-GPU Disaggregationcited by 0
  9. 2026FluidPD: In-Place Elasticity for SLO-Aware Prefill-Decode Disaggregated LLM Servingcited by 0