mlsanityData to 14 Oct 2026

Lightweight Scheduling Policies for Continuously Batched LLM Serving Across GPU Architectures

arXiv 2610.15899 ↗First posted 11 Oct 2026cs.DC cs.LGcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Large language model serving is increasingly limited not only by model size and GPU speed, but also by scheduling during autoregressive decoding, where continuous batching interleaves requests at token granularity as sequences arrive and finish. We study whether lightweight scheduling policies can improve latency-throughput tradeoffs without changing model architectures, kernels, or deploying learned schedulers. We implement a Python/PyTorch continuous-batching testbench with real Hugging Face execution and evaluate eight policies across 1,152 GPU-model-scheduler-workload runs using ShareGPT-derived traffic, six Qwen, Gemma, and Mistral models, and Modal L40S, A100, and H100 GPUs. Scheduler choice changes throughput, TTFT, completion latency, and tail behavior under fixed model, GPU, workload, batch size, and prefill budget. Aging shortest-prompt-first is the strongest practical baseline in many L40S and mixed-workload settings, while oracle remaining-length scheduling shows that job-size knowledge alone is insufficient because batching and hardware effects also shape performance. The custom dataset has been uploaded to Hugging Face.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 5 of its 8 references that are in the map. First posted on arXiv 11 Oct 2026.

Builds on

5 of 8 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Efficient LLM Scheduling by Learning to Rank28 Aug
  2. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  3. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills31 Aug
  4. 2023Fast Distributed Inference Serving for Large Language Models10 May

Other references · 1

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemption
  2. 2024ConServe: Fine-Grained GPU Harvesting for LLM Online and Offline Co-Serving
  3. 2025CascadeInfer: Length-Aware Scheduling of LLM Serving with Low Latency and Load Balancing
  4. 2026Understanding Efficiency: Quantization, Batching, and Serving Strategies in LLM Energy Use
  5. 2025Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs
  6. 2026SLIM: Saturation-Aware Lightweight Performance Modeling for LLM Serving
  7. 2025LeMix: Unified Scheduling for LLM Training and Inference on Multi-GPU Systems
  8. 2025WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Serving