mlsanityData to 7 Oct 2026

SageSched: Efficient LLM Scheduling Confronting Demand Uncertainty and Hybridity

arXiv 2603.07917 ↗First posted 9 Mar 2026cs.DCcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Efficient LLM inference scheduling is crucial for user experience. However, LLM inferences exhibit remarkable demand uncertainty (with unknown output length beforehand) and hybridity (being both compute and memory intensive). Existing LLM schedulers rely on simple heuristics or focus purely on compute resource, suffering suboptimal performance. In this work, we propose SageSched, an efficient LLM scheduler that properly handles demand uncertainty and hybridity of inference workloads. SageSched combines prompt contents with the past inference results to predict output-length distribution in a light-weight and also accurate manner. Meanwhile, it models the true service cost of an inference request with both compute and memory aspects considered. Finally, SageSched employs an uncertainty-aware scheduling policy that can yield the best overall efficiency given the request cost distributions. Testbed experiments over diverse setups confirm that SageSched can attain an efficiency improvement of over 28.7%.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 31 of its 49 references that are in the map. First posted on arXiv 9 Mar 2026.

Builds on

31 of 49 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025Past-Future Scheduler for LLM Serving under SLA Guarantees14 Jul
  2. 2024Efficient LLM Scheduling by Learning to Rank28 Aug
  3. 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool25 Jun
  4. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving24 Jun
  5. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving5 Jun
  6. 2024Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction12 Apr
  7. 2024MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving2 Apr
  8. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  9. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  10. 2023Fairness in Serving Large Language Models31 Dec
  11. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline22 May
  12. 2023Fast Distributed Inference Serving for Large Language Models10 May

Other references · 19

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  3. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  5. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  6. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  7. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  8. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  9. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  10. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  11. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 242
  12. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 120
7 more
  1. 2023Online Speculative Decodingcited by 109
  2. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
  3. 2024LLM3:Large Language Model-based Task and Motion Planning with Motion Failure Reasoningcited by 63
  4. 2025R-KV: Redundancy-aware KV Cache Compression for Reasoning Modelscited by 52
  5. 2024Don't Stop Me Now: Embedding Based Scheduling for LLMscited by 34
  6. 2025Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inferencecited by 32
  7. 2023TypeFly: Flying Drones with Large Language Modelcited by 23

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Prediction: Tail-Aware Scheduling for LLM Inference
  2. 2025Optimal Scheduling Algorithms for LLM Inference: Theory and Practice
  3. 2026Position: LLM Serving Needs Mathematical Optimization and Algorithmic Foundations, Not Just Heuristics
  4. 2025MIST: A Co-Design Framework for Heterogeneous, Multi-Stage LLM Inference
  5. 2025Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving
  6. 2026Geometry-Aware Online Scheduling for LLM Serving: From Theoretical Bound to System Practice
  7. 2026Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving
  8. 2025A Predictive and Synergistic Two-Layer Scheduling Framework for LLM Serving