mlsanityData to 7 Oct 2026

DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving

arXiv 2401.09670 ↗First posted 18 Jan 2024cs.DCcited by 595

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

DistServe improves the performance of large language models (LLMs) serving by disaggregating the prefill and decoding computation. Existing LLM serving systems colocate the two phases and batch the computation of prefill and decoding across all users and requests. We find that this strategy not only leads to strong prefill-decoding interferences but also couples the resource allocation and parallelism plans for both phases. LLM applications often emphasize individual latency for each phase: time to first token (TTFT) for the prefill phase and time per output token (TPOT) of each request for the decoding phase. In the presence of stringent latency requirements, existing systems have to prioritize one latency over the other, or over-provision compute resources to meet both. DistServe assigns prefill and decoding computation to different GPUs, hence eliminating prefill-decoding interferences. Given the application's TTFT and TPOT requirements, DistServe co-optimizes the resource allocation and parallelism strategy tailored for each phase. DistServe also places the two phases according to the serving cluster's bandwidth to minimize the communication caused by disaggregation. As a result, DistServe significantly improves LLM serving performance in terms of the maximum rate that can be served within both TTFT and TPOT constraints on each GPU. Our evaluations show that on various popular LLMs, applications, and latency requirements, DistServe can serve 7.4x more requests or 12.6x tighter SLO, compared to state-of-the-art systems, while staying within latency constraints for > 90% of requests.

Key facts. Cited by 595 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 18 of its 48 references that are in the map. First posted on arXiv 18 Jan 2024.

Builds on

18 of 48 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Fast Distributed Inference Serving for Large Language Models10 May

Other references · 17

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  7. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  8. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  9. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  10. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  11. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  12. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 516
5 more
  1. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
  2. 2020Serving DNNs like Clockwork: Performance Predictability from the Bottom Upcited by 119
  3. 2020TurboTransformers: An Efficient GPU Serving System For Transformer Modelscited by 71
  4. 2020Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learningcited by 68
  5. 2020FT-CNN: Algorithm-Based Fault Tolerance for Convolutional Neural Networkscited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 78; 2025: 202; 2026: 315.

595 papers.

Sort:

By area:

  1. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  2. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 211
  3. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  4. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 193
  5. 2025dKV-Cache: The Cache for Diffusion Language Modelscited by 140
  6. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 136
  7. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
  8. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  9. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 120
  10. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 119
  11. 2024Vidur: A Large-Scale Simulation Framework For LLM Inferencecited by 115
  12. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 114
38 more
  1. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelismcited by 110
  2. 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generationcited by 97
  3. 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 96
  4. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttentioncited by 93
  5. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 93
  6. 2024NanoFlow: Towards Optimal Large Language Model Serving Throughputcited by 85
  7. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systemscited by 84
  8. 2024Efficient LLM Scheduling by Learning to Rankcited by 83
  9. 2024Preble: Efficient Distributed Prompt Scheduling for LLM Servingcited by 68
  10. 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Poolcited by 67
  11. 2024Fast On-device LLM Inference with NPUscited by 58
  12. 2024Efficiently Scaling LLM Reasoning with Certaindexcited by 56
  13. 2024Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flowcited by 54
  14. 2025Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decodingcited by 52
  15. 2025The Best Instruction-Tuning Data are Those That Fitcited by 48
  16. 2024LLM Inference Serving: Survey of Recent Advances and Opportunitiescited by 46
  17. 2024POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inferencecited by 46
  18. 2024Post-Training Sparse Attention with Double Sparsitycited by 45
  19. 2024Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Servicescited by 44
  20. 2025Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Livecited by 43
  21. 2024Optimizing RLHF Training for Large Language Models with Stage Fusioncited by 41
  22. 2025KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Providercited by 41
  23. 2024SampleAttention: Near-Lossless Acceleration of Long Context LLM Inference with Adaptive Structured Sparse Attentioncited by 39
  24. 2025Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architecturescited by 38
  25. 2025The Cost of Dynamic Reasoning: Demystifying AI Agents and Test-Time Scaling from an AI Infrastructure Perspectivecited by 38
  26. 2024NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inferencecited by 37
  27. 2024Queue management for slo-oriented large language model servingcited by 36
  28. 2024ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocationcited by 34
  29. 2024Teola: Towards End-to-End Optimization of LLM-based Applicationscited by 32
  30. 2026LongCat-Flash-Thinking-2601 Technical Reportcited by 32
  31. 2025MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelismcited by 31
  32. 2025The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimizationcited by 31
  33. 2024M'elange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneitycited by 30
  34. 2024LLMServingSim: A HW/SW Co-Simulation Infrastructure for LLM Inference Serving at Scalecited by 30
  35. 2025ServeGen: Workload Characterization and Generation of Large Language Model Serving in Productioncited by 29
  36. 2025TAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platformscited by 28
  37. 2024P/D-Serve: Serving Disaggregated Large Language Model at Scalecited by 27
  38. 2024Context Parallelism for Scalable Million-Token Inferencecited by 25

50 most cited of 595 All 595 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025DOPD: A Dynamic PD-Disaggregation Architecture for Maximizing Goodput in LLM Inference Serving
  2. 2026SLO-Aware Compute Resource Allocation for Prefill-Decode Disaggregated LLM Inference
  3. 2025SeaLLM: Service-Aware and Latency-Optimized Resource Sharing for Large Language Model Inference
  4. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills
  5. 2026E2LLM: Towards Efficient LLM Serving in Heterogeneous Edge/Fog Environments
  6. 2025HexGen-2: Disaggregated Generative Inference of LLMs in Heterogeneous Environment
  7. 2025Characterizing Communication Patterns in Distributed Large Language Model Inference
  8. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache