mlsanityData to 14 Oct 2026

Stable Token-level Scheduling for Serverless LLM Inference

arXiv 2610.17689 ↗First posted 13 Oct 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Serverless inference is emerging as an efficient deployment model for Large Language Models (LLMs) due to its ability to elastically scale resources, improve GPU utilization through resource sharing, and reduce infrastructure costs. However, existing approaches primarily optimize time to first token (TTFT) while overlooking inter-token latency (ITL), a key determinant of user experience during response generation. We observe that LLM inference comprises a computation-intensive prefill stage and a bandwidth-bound decode stage, where admitting new requests can disrupt ongoing decoding and increase latency. To address this, we propose Token Window Pacing (TWP), a closed-loop scheduling framework that leverages runtime feedback from ITL, queue depth, and workload characteristics to regulate request admission at the token level instead of the batch level. TWP dynamically adjusts a selfregulating token window and incorporates chunk-based prefill to minimize interference from incoming requests. Experiments on realistic serverless workloads show that TWP delivers stable TTFT, significantly reduces latency spikes, lowers P99 tail latency, minimizes ITL variation, and improves throughput by up to 45% over ServerlessLLM and 58% over conventional schedulers. These results demonstrate that feedback-driven tokenlevel scheduling enables stable, high-throughput serverless LLM inference without modifying the underlying model.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 18 of its 31 references that are in the map. First posted on arXiv 13 Oct 2026.

Builds on

18 of 31 references.

References · 18

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  4. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  5. 2024Token-Budget-Aware LLM Reasoningcited by 247
  6. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 193
  7. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 177
  8. 2024ServerlessLLM: Low-Latency Serverless Inference for Large Language Modelscited by 80
  9. 2020Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Providercited by 66
  10. 2025DeepServe: Serverless Large Language Model Serving at Scalecited by 18
  11. 2025HydraServe: Minimizing Cold Start Latency for Serverless LLM Serving in Public Cloudscited by 14
  12. 2025TokenScale: Timely and Accurate Autoscaling for Disaggregated LLM Serving with Token Velocitycited by 12
6 more
  1. 2025λScale: Enabling Fast Scaling for Serverless Large Language Model Inferencecited by 8
  2. 2025Optimal Scheduling Algorithms for LLM Inference: Theory and Practicecited by 6
  3. 2025WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Servingcited by 5
  4. 2025Semi-Clairvoyant Scheduling of Speculative Decoding Requests to Minimize LLM Inference Latencycited by 4
  5. 2025Predictive-LoRA: A Proactive and Fragmentation-Aware Serverless Inference System for LLMscited by 3
  6. 2025Tangram: Accelerating Serverless LLM Loading through GPU Memory Reuse and Affinitycited by 2

Built on by

Not cited yet.