mlsanityData to 7 Oct 2026

Towards Load-Aware Prefill Deflection for Disaggregated LLM Serving

arXiv 2607.02043 ↗First posted 2 Jul 2026cs.DC cs.AIcited by 2

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Disaggregated LLM serving runs prefill and decode on separate GPU pools to keep the two phases from interfering. In practice, this creates a new asymmetry: under bursty, heavy-tailed workloads prefill nodes saturate while decode nodes have compute underutilized, and on a production-style A100 cluster with 2 prefill and 2 decode nodes (2P2D), we find that prefill execution accounts for only 2-23% of P95 Time-to-First-Token (TTFT). Queuing and inter-node GPU-GPU KV-cache transfer account for the rest. We present a proactive prefill-deflecting scheduler that lets decode nodes serve prefill phase of requests as chunked-prefill steps interleaved with their in-flight decode batches. For each queued request, we estimate the TTFT it would see on the prefill node, and on every decode node, search for the largest chunk schedule that keeps in-flight decodes within their Time-Between-Tokens (TBT) SLO and deflect when the decode path helps tail latency. Because the prefill phase of deflected requests runs in place on the decode node, the inter-node KV transfer is eliminated. Implemented on vLLM and evaluated on production-style traces with DeepSeek-V2-Lite, our approach reduces P95 TTFT by upto 81% and raises SLO attainment by upto 79% over state-of-the-art disaggregated schedulers, at sub-millisecond per-request routing cost.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 11 of its 15 references that are in the map. First posted on arXiv 2 Jul 2026.

Builds on

11 of 15 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Serving4 Aug
  2. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving24 Jun
  3. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar

Other references · 8

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  3. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  4. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  5. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  6. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 120
  7. 2024POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inferencecited by 46
  8. 2026Not All Prefills Are Equal: PPD Disaggregation for Multi-turn LLM Servingcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Unified AI Gateway: A Framework for Joint Model Routing and KV Cache Managementcited by 0
  2. 2026FluidPD: In-Place Elasticity for SLO-Aware Prefill-Decode Disaggregated LLM Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026StreamServe: Adaptive Speculative Flows for Low-Latency Disaggregated LLM Serving
  2. 2025DynaServe: Unified and Elastic Execution for Dynamic Disaggregated LLM Serving
  3. 2026Fairness-Aware and Latency-Controllable Scheduling for Chunked-Prefill LLM Serving
  4. 2025Nexus:Proactive Intra-GPU Disaggregation of Prefill and Decode in LLM Serving
  5. 2025Boosting LLM Serving through Spatial-Temporal GPU Resource Sharing
  6. 2025TokenScale: Timely and Accurate Autoscaling for Disaggregated LLM Serving with Token Velocity
  7. 2026FlowPrefill: Decoupling Preemption from Prefill Scheduling Granularity to Mitigate Head-of-Line Blocking in LLM Serving
  8. 2026LAPS: A Length-Aware-Prefill LLM Serving System