mlsanityData to 14 Oct 2026

Beyond Prediction: Tail-Aware Scheduling for LLM Inference

arXiv 2606.18431 ↗First posted 16 Jun 2026cs.LG cs.DCcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

LLM serving exhibits extreme length variability, making size-based scheduling difficult in practice. Recent LLM schedulers approximate SJF/SRPT using predicted decode lengths or ranks and primarily report mean-centric metrics such as TTFT and TBT. We show that these prediction-driven policies can be fragile under distribution shifts, bursty arrivals, and GPU memory pressure, while offering limited control over the tail latency (P90-P99) that dominates user experience, even with perfect decode-length knowledge. We introduce a distribution-aware, prediction-free scheduling framework that replaces explicit length prediction with soft priority boosting driven by lightweight statistical signals. Our design co-optimizes scheduling and cache-aware preemption to account for memory-coupled decode dynamics across workload mixes. Evaluated on production and open-source traces, our method reduces P99 TTLT by up to 35-50% relative to SRPT with perfect length knowledge and reduces TTFT by 34-47% across workloads, including reasoning-heavy and chat-heavy tasks. These results demonstrate a robust alternative for optimizing tail latency in online LLM serving.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 21 of its 44 references that are in the map. First posted on arXiv 16 Jun 2026.

Builds on

21 of 44 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025On Evaluating Performance of LLM Inference Serving Systems11 Jul
  2. 2025Nexus:Proactive Intra-GPU Disaggregation of Prefill and Decode in LLM Serving9 Jul
  3. 2025Throughput-Optimal Scheduling Algorithms for LLM Inference and AI Agents10 Apr
  4. 2024Efficient LLM Scheduling by Learning to Rank28 Aug
  5. 2024Queue management for slo-oriented large language model serving5 Jun
  6. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving5 Jun
  7. 2024Preble: Efficient Distributed Prompt Scheduling for LLM Serving8 May
  8. 2024Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Services25 Apr
  9. 2024Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction12 Apr
  10. 2024MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving2 Apr
  11. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan

Other references · 10

  1. 2025s1: Simple test-time scalingcited by 1,341
  2. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
  3. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  4. 2025Autellix: An Efficient Serving Engine for LLM Agents as General Programscited by 58
  5. 2025Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inferencecited by 53
  6. 2024Don't Stop Me Now: Embedding Based Scheduling for LLMscited by 35
  7. 2024No Request Left Behind: Tackling Heterogeneity in Long-Context LLM Inference with Medhacited by 13
  8. 2025Kairos: Low-latency Multi-Agent Serving with Shared LLMs and Excessive Loads in the Public Cloudcited by 11
  9. 2025Niyama : Breaking the Silos of LLM Inference Servingcited by 7
  10. 2025Tail-Optimized Caching for LLM Inferencecited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026SageSched: Efficient LLM Scheduling Confronting Demand Uncertainty and Hybridity
  2. 2026Geometry-Aware Online Scheduling for LLM Serving: From Theoretical Bound to System Practice
  3. 2025Optimal Scheduling Algorithms for LLM Inference: Theory and Practice
  4. 2026Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving
  5. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
  6. 2025DynaServe: Unified and Elastic Execution for Dynamic Disaggregated LLM Serving
  7. 2025Optimizing LLM Inference: Fluid-Guided Online Scheduling with Memory Constraints
  8. 2025Ranking Before Serving: Low-Latency LLM Serving via Pairwise Learning-to-Rank