Staggered Batch Scheduling: Co-optimizing Time-to-First-Token and Throughput for High-Efficiency LLM Inference
Not on the map. Why?
Abstract (arXiv)
The evolution of Large Language Model (LLM) serving towards complex, distributed architectures--specifically the P/D-separated, large-scale DP+EP paradigm--introduces distinct scheduling challenges. Unlike traditional deployments where schedulers can treat instances as black boxes, DP+EP architectures exhibit high internal synchronization costs. We identify that immediate request dispatching in such systems leads to severe in-engine queuing and parallelization bubbles, degrading Time-to-First-Token (TTFT). To address this, we propose Staggered Batch Scheduling (SBS), a mechanism that deliberately buffers requests to form optimal execution batches. This temporal decoupling eliminates internal queuing bubbles without compromising throughput. Furthermore, leveraging the scheduling window created by buffering, we introduce a Load-Aware Global Allocation strategy that balances computational load across DP units for both Prefill and Decode phases. Deployed on a production H800 cluster serving Deepseek-V3, our system reduces TTFT by 30%-40% and improves throughput by 15%-20% compared to state-of-the-art immediate scheduling baselines.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 14 of its 16 references that are in the map. First posted on arXiv 18 Dec 2025.
Builds on
14 of 16 references.
References · 14
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
- 2022Training Compute-Optimal Large Language Modelscited by 2,927
- 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
- 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
- 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
- 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
- 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
- 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
- 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 167
- 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 120
- 2024Preble: Efficient Distributed Prompt Scheduling for LLM Servingcited by 68
2 more
Built on by
Citing papers per year: 2026: 2.
2 papers.