mlsanityData to 14 Oct 2026

Deadline-Aware Adaptive Prefill Chunking for Efficient Large Language Model Serving

arXiv 2609.07883 ↗First posted 7 Sep 2026cs.CLcited by 2

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Continuous batching improves large language model (LLM) serving throughput, but long prompt prefills can delay decode iterations and violate inter-token latency objectives. Chunked prefill mitigates this interference, yet its chunk size is normally fixed: small chunks protect decode latency but repeatedly pay launch overhead, while large chunks improve prefill efficiency but create latency spikes. We introduce SLOWeave, an online scheduling method that selects the largest prefill chunk predicted to finish before the earliest active decode deadline. The decision requires no workload-specific chunk-size tuning and is computed by a logarithmic-time search over a monotone iteration-cost model. We prove that, whenever a decode-only iteration is feasible and the cost predictor is accurate, SLOWeave maximizes immediate prefill progress among decisions that preserve every active request's next-token deadline. We evaluate the method in a reproducible event-driven simulator and an iteration-level GPU runtime across chat, mixed-context, long-context, and bursty workloads. Under a 25ms time-per-output-token objective, SLOWeave improves goodput over the strongest fixed-chunk baseline by 39% on mixed requests and 38% on long-context requests. Under a stricter 10ms objective, the gains rise to 3.3× and 2.4×, respectively. These results isolate adaptive chunk sizing as a useful serving primitive and provide an implementation-ready controller for integration with iteration-level LLM runtimes.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 19 of its 24 references that are in the map. First posted on arXiv 7 Sep 2026.

Builds on

19 of 24 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving24 Jun
  2. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention7 May
  3. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism15 Apr
  4. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  5. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan

Other references · 14

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  3. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  4. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  5. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  6. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 258
  7. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 203
  8. 2019MLPerf Inference Benchmarkcited by 199
  9. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 152
  10. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 121
  11. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  12. 2024ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partitioncited by 82
2 more
  1. 2024D'ej`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Servingcited by 48
  2. 2025Long-Context Modeling with Dynamic Hierarchical Sparse Attention for Memory-Constrained LLM Inferencecited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Decode-Latency Feedback Prefill: A Model-Free Controller and Its Generalization Limitscited by 0
  2. 2026Scaling LLM Inference via Parallelism: Throughput-Latency Trade-offcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Fairness-Aware and Latency-Controllable Scheduling for Chunked-Prefill LLM Serving
  2. 2025LLM Serving Optimization with Variable Prefill and Decode Lengths
  3. 2025From Tokens to Layers: Redefining Stall-Free Scheduling for MoE Serving with Layered Prefill
  4. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills
  5. 2025Optimizing Long-context LLM Serving via Fine-grained Sequence Parallelism
  6. 2026FlowPrefill: Decoupling Preemption from Prefill Scheduling Granularity to Mitigate Head-of-Line Blocking in LLM Serving
  7. 2026An Interpretable Latency Model for Speculative Decoding in LLM Serving
  8. 2026MAPS: Memory-Aware Predictive Scheduling Framework for Large Language Model Serving