mlsanityData to 14 Oct 2026

CascadeInfer: Length-Aware Scheduling of LLM Serving with Low Latency and Load Balancing

arXiv 2512.19179 ↗First posted 22 Dec 2025cs.DCcited by 1

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Efficiently harnessing GPU compute is critical to improving user experience and reducing operational costs in large language model (LLM) services. However, current inference engine schedulers overlook the attention backend's sensitivity to request-length heterogeneity within a batch. As state-of-the-art models now support context windows exceeding 128K tokens, this once-tolerable inefficiency has escalated into a primary system bottleneck, causing severe performance degradation through GPU underutilization and increased latency. We present CascadeInfer, a runtime system that dynamically reschedules requests across multiple instances serving the same LLM to mitigate per-instance length heterogeneity. CascadeInfer partitions these instances into length-specialized groups, each handling requests within a designated length range, naturally forming a pipeline as requests flow through them. CascadeInfer devises a dynamic programming algorithm to efficiently find the stage partition with the best QoE, employs runtime range refinement together with decentralized load (re)balance both across and within groups, achieving a balanced and efficient multi-instance service. Our evaluation shows that, under the same configuration, CascadeInfer reduces end-to-end latency by up to 67% and tail latency by up to 69%, while improving overall system throughput by up to 2.89 times compared to the state-of-the-art multi-instance scheduling systems.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 17 of its 27 references that are in the map. First posted on arXiv 22 Dec 2025.

Builds on

17 of 27 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Locality-aware Fair Scheduling in LLM Serving24 Jan
  2. 2024Efficient LLM Scheduling by Learning to Rank28 Aug
  3. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving5 Jun
  4. 2024Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction12 Apr
  5. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  6. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline22 May
  7. 2023Fast Distributed Inference Serving for Large Language Models10 May

Other references · 10

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  4. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  5. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  6. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
  7. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 258
  8. 2024ServerlessLLM: Low-Latency Serverless Inference for Large Language Modelscited by 80
  9. 2025MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelismcited by 32
  10. 2025AIBrix: Towards Scalable, Cost-Effective Large Language Model Inference Infrastructurecited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026FleetOpt: Analytical Fleet Provisioning for LLM Inference with Compress-and-Route as Implementation Mechanismcited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2026Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving
  2. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache
  3. 2025LeMix: Unified Scheduling for LLM Training and Inference on Multi-GPU Systems
  4. 2025Parallax: Efficient LLM Inference Service over Decentralized Environment
  5. 2024AQUA: Network-Accelerated Memory Offloading for LLMs in Scale-Up GPU Domains
  6. 2026Scaling LLM Inference Beyond Amdahl`s Limits via Eliminating Non-Scalable Overheads ↗
  7. 2026Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemption
  8. 2025Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving