STAR: Decode-Phase Rescheduling for LLM Inference
Not on the map. Why?
Abstract (arXiv)
Large Language Model (LLM) inference has emerged as a fundamental paradigm, however, variations in output length cause severe workload imbalance in the decode phase, particularly for long-output reasoning tasks. Existing systems, such as PD disaggregation architectures, rely on static prefill-to-decode scheduling, which often results in SLO violations and OOM failures under evolving decode workloads. In this paper, we propose STAR, a decode rescheduling system powered by length prediction to anticipate future workloads. Our core contributions include: (1) A lightweight and continuous LLM-native prediction method that leverages LLM hidden state to model remaining generation length with high precision (reducing MAE by 49.42%) and low overhead (cutting predictor parameters by 93.28%); (2) A rescheduling solution in decode phase with a dynamic balancing mechanism that integrates current and predicted workloads, reducing P99 TPOT by 75.1% and achieving 2.63 times higher goodput.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 17 of its 40 references that are in the map. First posted on arXiv 15 Oct 2025.
Builds on
17 of 40 references.
References · 17
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2019The Curious Case of Neural Text Degenerationcited by 2,916
- 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
- 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
- 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
- 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 370
- 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 331
- 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 163
- 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 121
- 2023SpotServe: Serving Generative Large Language Models on Preemptible Instancescited by 87
- 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipelinecited by 86
5 more
- 2024Efficient LLM Scheduling by Learning to Rankcited by 85
- 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
- 2024Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Predictioncited by 66
- 2024Slice-Level Scheduling for High Throughput and Load Balanced LLM Servingcited by 13
- 2025FlowKV: A Disaggregated Inference Framework with Low-Latency KV Cache Transfer and Load-Aware Schedulingcited by 10
Built on by
Citing papers per year: 2026: 1.
1 paper.