Stage-Aware Communication Scheduling for Disaggregated LLM Serving
On the map: Efficient LLM inference and serving → Serving systems and request scheduling
Abstract (arXiv)
Meeting stringent Time-To-First-Token (TTFT) requirements is crucial for LLM applications. To improve efficiency, modern LLM serving systems adopt disaggregated architectures with diverse parallelisms, introducing complex multi-stage workflows involving reusable KV-block retrieval, collective communication, and P2D transfer. Flows from dependent stages overlap within and across requests on shared bottleneck links, making TTFT highly susceptible to network contention and necessitating stage-aware scheduling. Unfortunately, most existing works schedule flows in a stage-agnostic manner, leading to uncoordinated contention that constitutes a primary cause of SLO violations. In this paper, we present Nuska, a holistic multi-stage flow scheduling mechanism designed to maximize TTFT SLO attainment. At its core, Nuska approximates the Least-Laxity-First (LLF) scheduling policy without requiring precise knowledge of a request's remaining slack. It achieves this through a Defer-and-Promote principle implemented through a Reverse Multi-Level Queue (RMLQ) structure. By dynamically promoting task precedence as effective laxity diminishes, Nuska prioritizes flows with less laxity while preventing requests with loose SLOs from prematurely consuming network bandwidth. We implement Nuska as a pluggable module integrated into vLLM, and evaluate it on an 8-server, 32-GPU testbed as well as through large-scale simulations. Our results demonstrate that Nuska effectively outperforms state-of-the-art baselines, improving the TTFT SLO attainment by 1.2x-2.4x.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 31 of its 98 references that are in the map. First posted on arXiv 18 Mar 2026.
Builds on
31 of 98 references.
Closest ancestors · 6
Same direction, newest first
- 2026DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference25 Feb
- 2025FlowKV: A Disaggregated Inference Framework with Low-Latency KV Cache Transfer and Load-Aware Scheduling3 Apr
- 2025ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environments13 Feb
- 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool25 Jun
- 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving24 Jun
- 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism15 Apr
Other references · 25
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
- 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
- 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
- 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
- 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
- 2023Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agentscited by 333
- 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 223
- 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 203
- 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 121
- 2024Vidur: A Large-Scale Simulation Framework For LLM Inferencecited by 117
- 2022Accelerating Distributed MoE Training and Inference with Linacited by 109
13 more
- 2023Can ChatGPT replace StackOverflow? A Study on Robustness and Reliability of Large Language Model Code Generationcited by 73
- 2026Qwen3-Coder-Next Technical Reportcited by 71
- 2021TACCL: Guiding Collective Algorithm Synthesis using Communication Sketchescited by 38
- 2023CASSINI: Network-Aware Job Scheduling in Machine Learning Clusterscited by 26
- 2024Context Parallelism for Scalable Million-Token Inferencecited by 26
- 2025Triton-distributed: Programming Overlapping Kernels on Distributed AI Systems with the Triton Compilercited by 25
- 2025TileLink: Generating Efficient Compute-Communication Overlapping Kernels using Tile-Centric Primitivescited by 24
- 2025SLOs-Serve: Optimized Serving of Multi-SLO LLMscited by 19
- 2025TraCT: Disaggregated LLM Serving with CXL Shared Memory KV Cache at Rack-Scalecited by 13
- 2025MixNet: A Runtime Reconfigurable Optical-Electrical Fabric for Distributed Mixture-of-Experts Trainingcited by 7
- 2025Toward Low-Latency End-to-End Voice Agents for Telecommunications Using Streaming ASR, Quantized LLMs, and Real-Time TTScited by 5
- 2025Enhancing reliability in AI inference services: An empirical study on real production incidentscited by 5
- 2025AccelGen: Heterogeneous SLO-Guaranteed High-Throughput LLM Inference Serving for Diverse Applicationscited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.
- 2026MOSAIC: Masked Outsourcing of Secure AI Computationscited by 1
- 2026MOSAIC: Masked Outsourcing of Secure AI Computations31 Jul 2026
Similar, not cited
Close in wording, no citation either way
- 2026FlowPrefill: Decoupling Preemption from Prefill Scheduling Granularity to Mitigate Head-of-Line Blocking in LLM Serving
- 2026P-PAS: Prefill-Pressure Adaptive Scheduling for Long-Context LLM Serving
- 2026Taming Request Imbalance: SLO-Aware Scheduling for Disaggregated LLM Inference
- 2026Tropical: Enhancing SLO Attainment in Disaggregated LLM Serving via SLO-Aware Multiplexing
- 2025ProServe: Unified Multi-Priority Request Scheduling for LLM Serving
- 2025Ascendra: Dynamic Request Prioritization for Efficient LLM Serving
- 2025Optimal Scheduling Algorithms for LLM Inference: Theory and Practice
- 2026LAPS: A Length-Aware-Prefill LLM Serving System