# DynaServe: Unified and Elastic Execution for Dynamic Disaggregated LLM Serving

arXiv: https://arxiv.org/abs/2504.09285 · HTML: https://mlsanity.com/paper/2504.09285/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving.
- Builds on 19 of its 32 references that are in the map.
- First posted on arXiv 12 Apr 2025.

## Placed in

- Efficient LLM inference and serving / [Serving systems and request scheduling](https://mlsanity.com/inference/serving-systems-and-request-scheduling/)

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving](https://mlsanity.com/paper/2401.09670/) (2024)
- [Splitwise: Efficient generative LLM inference using phase splitting](https://mlsanity.com/paper/2311.18677/) (2023)
- [Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve](https://mlsanity.com/paper/2403.02310/) (2024)
- [A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents](https://mlsanity.com/paper/1804.05685/) (2018)
- [Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving](https://mlsanity.com/paper/2407.00079/) (2024)
- [SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills](https://mlsanity.com/paper/2308.16369/) (2023)
- [Llumnix: Dynamic Scheduling for Large Language Model Serving](https://mlsanity.com/paper/2406.03243/) (2024)
- [LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism](https://mlsanity.com/paper/2404.09526/) (2024)
- [NanoFlow: Towards Optimal Large Language Model Serving Throughput](https://mlsanity.com/paper/2408.12757/) (2024)
- [S3: Increasing GPU Utilization during Generative Inference for Higher Throughput](https://mlsanity.com/paper/2306.06000/) (2023)
- [Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction](https://mlsanity.com/paper/2404.08509/) (2024)
- [POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inference](https://mlsanity.com/paper/2410.18038/) (2024)
- [Apparate: Rethinking Early Exits to Tame Latency-Throughput Tensions in ML Serving](https://mlsanity.com/paper/2312.05385/) (2023)
- [Injecting Adrenaline into LLM Serving: Boosting Resource Utilization and Throughput via Attention Disaggregation](https://mlsanity.com/paper/2503.20552/) (2025)
- [Automatic Prompt Selection for Large Language Models](https://mlsanity.com/paper/2404.02717/) (2024)
- [Niyama : Breaking the Silos of LLM Inference Serving](https://mlsanity.com/paper/2503.22562/) (2025)

## Built on by (most cited first, 6 of 6)

- [Arrow: Adaptive Scheduling Mechanisms for Disaggregated LLM Inference Architecture](https://mlsanity.com/paper/2505.11916/) (2025)
- [RelayGR: Scaling Long-Sequence Generative Recommendation via Cross-Stage Relay-Race Inference](https://mlsanity.com/paper/2601.01712/) (2026)
- [Rethinking Latency Denial-of-Service: Attacking the LLM Serving Framework, Not the Model](https://mlsanity.com/paper/2602.07878/) (2026)
- [Lodestar: An Online-Learning LLM Inference Router](https://mlsanity.com/paper/2606.00946/) (2026)
- [LLM Inference on IMC-NoC Architecture with Balanced Dataflow and Fine-Grained Parallelism](https://mlsanity.com/paper/2609.00857/) (2026)
- [Scaling LLM Inference via Parallelism: Throughput-Latency Trade-off](https://mlsanity.com/paper/2610.19405/) (2026)

## Abstract (arXiv)

LLM inference must meet strict latency SLOs (e.g., 100 ms P99 time-between-tokens) while maximizing goodput. Yet, real-world variability in prompt and response lengths skews compute-intensive prefill and memory-bound decode phases, making both colocated (even with chunked prefill) and disaggregated deployments unable to simultaneously deliver low tail latency and high throughput. We introduce DynaServe, a high-performance LLM serving system built atop vLLM that unifies and extends both paradigms for maximizing goodput under SLO constraints, when handling unbalanced and dynamic workloads. It relies on a micro-request abstraction, which arbitrarily splits each request at any token boundary into at most two cooperating segments. A two-level scheduling framework then balances micro-request load across unified GPU instances. The global scheduler rapidly selects per-request split points by considering both the request's prefill/decode time ratio and the current load across GPU instances. The local schedulers on each GPU instance independently form SLO-aware batches, adjusting their composition in response to workload fluctuations, potential latency spikes and per-GPU under/over utilization. On real-world traces, DynaServe boosts the overall serving capacity from 1.15× to 3.07×, improves goodput by up to 1.91× and 1.61×, and improves the performance by up to 60% in a hybrid workload under SLO compared to state-of-the-art colocated and disaggregated baselines.