mlsanityData to 7 Oct 2026

DynaServe: Unified and Elastic Execution for Dynamic Disaggregated LLM Serving

arXiv 2504.09285 ↗First posted 12 Apr 2025cs.DCcited by 5

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

LLM inference must meet strict latency SLOs (e.g., 100 ms P99 time-between-tokens) while maximizing goodput. Yet, real-world variability in prompt and response lengths skews compute-intensive prefill and memory-bound decode phases, making both colocated (even with chunked prefill) and disaggregated deployments unable to simultaneously deliver low tail latency and high throughput. We introduce DynaServe, a high-performance LLM serving system built atop vLLM that unifies and extends both paradigms for maximizing goodput under SLO constraints, when handling unbalanced and dynamic workloads. It relies on a micro-request abstraction, which arbitrarily splits each request at any token boundary into at most two cooperating segments. A two-level scheduling framework then balances micro-request load across unified GPU instances. The global scheduler rapidly selects per-request split points by considering both the request's prefill/decode time ratio and the current load across GPU instances. The local schedulers on each GPU instance independently form SLO-aware batches, adjusting their composition in response to workload fluctuations, potential latency spikes and per-GPU under/over utilization. On real-world traces, DynaServe boosts the overall serving capacity from 1.15× to 3.07×, improves goodput by up to 1.91× and 1.61×, and improves the performance by up to 60% in a hybrid workload under SLO compared to state-of-the-art colocated and disaggregated baselines.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 19 of its 32 references that are in the map. First posted on arXiv 12 Apr 2025.

Builds on

19 of 32 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025Injecting Adrenaline into LLM Serving: Boosting Resource Utilization and Throughput via Attention Disaggregation26 Mar
  2. 2024NanoFlow: Towards Optimal Large Language Model Serving Throughput22 Aug
  3. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving24 Jun
  4. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving5 Jun
  5. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism15 Apr
  6. 2024Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction12 Apr
  7. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  8. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  9. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills31 Aug

Other references · 10

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  5. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 477
  6. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
  7. 2024POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inferencecited by 46
  8. 2023Apparate: Rethinking Early Exits to Tame Latency-Throughput Tensions in ML Servingcited by 15
  9. 2024Automatic Prompt Selection for Large Language Modelscited by 9
  10. 2025Niyama : Breaking the Silos of LLM Inference Servingcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2025Arrow: Adaptive Scheduling Mechanisms for Disaggregated LLM Inference Architecturecited by 5
  2. 2026RelayGR: Scaling Long-Sequence Generative Recommendation via Cross-Stage Relay-Race Inferencecited by 2
  3. 2026Rethinking Latency Denial-of-Service: Attacking the LLM Serving Framework, Not the Modelcited by 2
  4. 2026Lodestar: An Online-Learning LLM Inference Routercited by 0
  5. 2026LLM Inference on IMC-NoC Architecture with Balanced Dataflow and Fine-Grained Parallelismcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026RAPID-Serve: Resource-efficient and Accelerated P/D Intra-GPU Disaggregation
  2. 2024A System for Microserving of LLMs
  3. 2026LLM-Emu: Native Runtime Emulation of LLM Inference via Profile-Driven Sampling
  4. 2026AlignedServe: Orchestrating Prefix-aware Batching to Build a High-throughput and Computing-efficient LLM Serving System
  5. 2026DeltaServe: Host-Agnostic Co-Serving of Inference and Fine-Tuning for LLMs
  6. 2025Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving
  7. 2026Scaling LLM Inference Beyond Amdahl`s Limits via Eliminating Non-Scalable Overheads ↗
  8. 2025DOPD: A Dynamic PD-Disaggregation Architecture for Maximizing Goodput in LLM Inference Serving