mlsanityData to 7 Oct 2026

FlowPrefill: Decoupling Preemption from Prefill Scheduling Granularity to Mitigate Head-of-Line Blocking in LLM Serving

arXiv 2602.16603 ↗First posted 18 Feb 2026cs.DC cs.AIcited by 3

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

The growing demand for large language models (LLMs) requires serving systems to handle many concurrent requests with diverse service level objectives (SLOs). This exacerbates head-of-line (HoL) blocking during the compute-intensive prefill phase, where long-running requests monopolize resources and delay higher-priority ones, leading to widespread time-to-first-token (TTFT) SLO violations. While chunked prefill enables interruptibility, it introduces an inherent trade-off between responsiveness and throughput: reducing chunk size improves response latency but degrades computational efficiency, whereas increasing chunk size maximizes throughput but exacerbates blocking. This necessitates an adaptive preemption mechanism. However, dynamically balancing execution granularity against scheduling overheads remains a key challenge. In this paper, we propose FlowPrefill, a TTFT-goodput-optimized serving system that resolves this conflict by decoupling preemption granularity from scheduling frequency. To achieve adaptive prefill scheduling, FlowPrefill introduces two key innovations: 1) Operator-Level Preemption, which leverages operator boundaries to enable fine-grained execution interruption without the efficiency loss associated with fixed small chunking; and 2) Event-Driven Scheduling, which triggers scheduling decisions only upon request arrival or completion events, thereby supporting efficient preemption responsiveness while minimizing control-plane overhead. Evaluation on real-world production traces shows that FlowPrefill improves maximum goodput by up to 5.6× compared to state-of-the-art systems while satisfying heterogeneous SLOs.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 25 of its 49 references that are in the map. First posted on arXiv 18 Feb 2026.

Builds on

25 of 49 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Past-Future Scheduler for LLM Serving under SLA Guarantees14 Jul
  2. 2025semi-PD: Towards Efficient LLM Serving via Phase-Wise Disaggregated Computation and Unified Storage28 Apr
  3. 2024Efficient LLM Scheduling by Learning to Rank28 Aug
  4. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving24 Jun
  5. 2024Queue management for slo-oriented large language model serving5 Jun
  6. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  7. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan

Other references · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2025Qwen3 Technical Reportcited by 8,195
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  8. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  9. 2024Qwen2.5-Coder Technical Reportcited by 1,151
  10. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  11. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  12. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 120
6 more
  1. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
  2. 2020TurboTransformers: An Efficient GPU Serving System For Transformer Modelscited by 71
  3. 2025KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Providercited by 41
  4. 2024Don't Stop Me Now: Embedding Based Scheduling for LLMscited by 34
  5. 2024Efficiently Serving Large Multimodal Models Using EPD Disaggregationcited by 16
  6. 2025From Tokens to Layers: Redefining Stall-Free Scheduling for MoE Serving with Layered Prefillcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Stateful Worlds, Stateless Elasticity: Exact-State Serving for Interactive World Modelscited by 0
  2. 2026P-PAS: Prefill-Pressure Adaptive Scheduling for Long-Context LLM Servingcited by 0
  3. 2026Beyond Binary Priorities: Multi-Tier SLA Scheduling for Large Language Model Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Fairness-Aware and Latency-Controllable Scheduling for Chunked-Prefill LLM Serving
  2. 2026LAPS: A Length-Aware-Prefill LLM Serving System
  3. 2026Towards Load-Aware Prefill Deflection for Disaggregated LLM Serving
  4. 2026Stage-Aware Communication Scheduling for Disaggregated LLM Serving
  5. 2026Deadline-Aware Adaptive Prefill Chunking for Efficient Large Language Model Serving
  6. 2026Taming Request Imbalance: SLO-Aware Scheduling for Disaggregated LLM Inference ↗
  7. 2026FluidPD: In-Place Elasticity for SLO-Aware Prefill-Decode Disaggregated LLM Serving
  8. 2026MAPS: Memory-Aware Predictive Scheduling Framework for Large Language Model Serving