mlsanityData to 14 Oct 2026

A Predictive and Synergistic Two-Layer Scheduling Framework for LLM Serving

arXiv 2509.23384 ↗First posted 27 Sep 2025cs.DCcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

LLM inference serving typically scales out with a two-tier architecture: a cluster router distributes requests to multiple inference engines, each of which then in turn performs its own internal scheduling. However, this commonly used paradigm suffers from critical, systemic inefficiency caused by the information gaps across two layers. At the cluster-layer, the router mainly relies on lagging, coarse-grained metrics, such as average latency and queue length to make decisions, resulting in "decision lag" that leads to suboptimal request routing. At the engine-layer, static heuristic scheduling policies cannot effectively handle the dynamic workloads, leading a poor balance between latency and throughput. Besides, these gaps may cause SLO violations and resource waste, especially in heterogeneous cloud environments. To bridge such gaps, we propose NexusSched, a cross-layer framework that shifts LLM serving system from reactive load balancing to predictive orchestration. The core of NexusSched lies in a structurally-informed online performance model that provides accurate, forward-looking per-step latency and capacity estimations. This model empowers two key components. At the engine-layer, LENS performs SLO-aware, adaptive scheduling, dynamically optimizing batching to meet SLOs under real-time loads. At the cluster-layer, PRISM uses predictive signals to perform state-driven routing, maximizing cluster-wide performance and SLO attainment. Performance evaluations show that NexusSched improves SLO attainment by 43% on average and achieves up to 3x throughput speedup in long-context and heterogeneous scenarios. Besides, we also deploy NexusSched on FlowGPT's clusters to demonstrate its advantages in production environment.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 25 of its 40 references that are in the map. First posted on arXiv 27 Sep 2025.

Builds on

25 of 40 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving5 Jun
  2. 2024Aladdin: Joint Placement and Scaling for SLO-Aware LLM Serving11 May
  3. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  4. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  5. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instances27 Nov

Other references · 20

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  4. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  5. 2024A Survey on Large Language Models for Code Generationcited by 658
  6. 2020Recipes for building an open-domain chatbotcited by 593
  7. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 527
  8. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
  9. 2020Towards a Human-like Open-Domain Chatbotcited by 470
  10. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 258
  11. 2023Planning with Large Language Models for Code Generationcited by 182
  12. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 112
8 more
  1. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  2. 2024ExeGPT: Constraint-Aware Resource Scheduling for LLM Inferencecited by 45
  3. 2025Generalization Bias in Large Language Model Summarization of Scientific Researchcited by 30
  4. 2025SLOs-Serve: Optimized Serving of Multi-SLO LLMscited by 19
  5. 2025AIBrix: Towards Scalable, Cost-Effective Large Language Model Inference Infrastructurecited by 13
  6. 2025SkyWalker: A Locality-Aware Cross-Region Load Balancer for LLM Inferencecited by 10
  7. 2025SLO-Aware Scheduling for Large Language Model Inferencescited by 7
  8. 2025FineServe: Precision-Aware KV Slab and Two-Level Scheduling for Heterogeneous Precision LLM Servingcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Niyama : Breaking the Silos of LLM Inference Serving
  2. 2024Intelligent Router for LLM Workloads: Improving Performance Through Workload-Aware Load Balancing
  3. 2026Position: LLM Serving Needs Mathematical Optimization and Algorithmic Foundations, Not Just Heuristics
  4. 2026RouteBalance: Fused Model Routing and Load Balancing for Heterogeneous LLM Serving
  5. 2026SageSched: Efficient LLM Scheduling Confronting Demand Uncertainty and Hybridity
  6. 2026Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving
  7. 2026Beyond Binary Priorities: Multi-Tier SLA Scheduling for Large Language Model Serving
  8. 2025DynaServe: Unified and Elastic Execution for Dynamic Disaggregated LLM Serving