mlsanityData to 7 Oct 2026

TokenScale: Timely and Accurate Autoscaling for Disaggregated LLM Serving with Token Velocity

arXiv 2512.03416 ↗First posted 3 Dec 2025cs.DCcited by 9

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

The architectural shift to prefill/decode (PD) disaggregation in LLM serving improves resource utilization but struggles with the bursty nature of modern workloads. Existing autoscaling policies, often retrofitted from monolithic systems like those in AIBrix and DistServe, rely on lagging indicators such as GPU utilization or coarse-grained request counts. This results in slow reactions to load spikes, leading to significant Time-to First-Token (TTFT) and Time-Per-Output-Token (TPOT) SLO violations and costly over-provisioning. We introduce TokenScale, an autoscaling framework that resolves this performance mismatch through two innovations. First, we propose Token Velocity, a novel metric that unifies the prefill, network, and decode stages by quantifying their rate of work. As a leading indicator of system backpressure, it enables proactive scaling. Second, Convertible Decoders allow decoder GPUs to dynamically execute prefill tasks during traffic spikes, creating a rapid-response buffer that absorbs bursts and eliminates the initialization latency of new prefillers. Our evaluation on a GPU cluster with production traces shows TokenScale improves SLO attainment from 50-88% to 80-96% and reduces costs by 4-14% over state-of-the-art systems, including DistServe, BlitzScale, and AIBrix. By uniting a predictive metric with a flexible system design, TokenScale significantly boosts the performance and efficiency of disaggregated LLM serving infrastructure.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 28 of its 47 references that are in the map. First posted on arXiv 3 Dec 2025.

Builds on

28 of 47 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025HFX: Joint Design of Algorithms and Systems for Multi-SLO Serving and Fast Scaling21 Aug
  2. 2025MorphServe: Efficient and Workload-Aware LLM Serving via Runtime Quantized Layer Swapping and KV Cache Resizing24 May
  3. 2025FlowKV: A Disaggregated Inference Framework with Low-Latency KV Cache Transfer and Load-Aware Scheduling3 Apr
  4. 2025PipeBoost: Resilient Pipelined Architecture for Fast Serverless LLM Scaling22 Mar
  5. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving24 Jun
  6. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving5 Jun
  7. 2024MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving2 Apr
  8. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  9. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems31 Jan
  10. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan

Other references · 18

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  3. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 119
  4. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 106
  5. 2024ServerlessLLM: Low-Latency Serverless Inference for Large Language Modelscited by 78
  6. 2020Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Providercited by 65
  7. 2024BLITZSCALE: Fast and Live Large Model Autoscaling with O(1) Host Cachingcited by 23
  8. 2025PrefillOnly: An Inference Engine for Prefill-only Workloads in Large Language Model Applicationscited by 21
  9. 2025DeepServe: Serverless Large Language Model Serving at Scalecited by 17
  10. 2025SLOs-Serve: Optimized Serving of Multi-SLO LLMscited by 17
  11. 2025AIBrix: Towards Scalable, Cost-Effective Large Language Model Inference Infrastructurecited by 12
  12. 2023How Does It Function? Characterizing Long-term Trends in Production Serverless Workloadscited by 10
6 more
  1. 2025SkyWalker: A Locality-Aware Cross-Region Load Balancer for LLM Inferencecited by 9
  2. 2025λScale: Enabling Fast Scaling for Serverless Large Language Model Inferencecited by 7
  3. 2025PolyServe: Efficient Multi-SLO Serving at Scalecited by 5
  4. 2025EcoServe: Enabling Cost-effective LLM Serving with Proactive Intra- and Inter-Instance Orchestrationcited by 4
  5. 2025Dilu: Enabling GPU Resourcing-on-Demand for Serverless DL Serving via Introspective Elasticitycited by 3
  6. 2025Towards Resource-Efficient Serverless LLM Inference with SLINFERcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 9.

9 papers.

  1. 2026FLYING SERVING: On-the-Fly Parallelism Switching for Large Language Model Servingcited by 5
  2. 2026Simple is Better: Multiplication May Be All You Need for LLM Request Schedulingcited by 2
  3. 2026Observation, Not Prediction: Conversation-Level Disaggregated Scheduling for Agentic Servingcited by 2
  4. 2026KAIROS: Stateful, Context-Aware Power-Efficient Agentic Inference Servingcited by 1
  5. 2026FASER: Fine-Grained Phase Management for Speculative Decoding in Dynamic LLM Servingcited by 1
  6. 2026Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference servingcited by 0
  7. 2026Analytical Power-Aware Provisioning for Prefill-Decode Disaggregated AI Inferencecited by 0
  8. 2026Cross-Model Autoscaling for Shared LLM Servingcited by 0
  9. 2026FluidPD: In-Place Elasticity for SLO-Aware Prefill-Decode Disaggregated LLM Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Towards Load-Aware Prefill Deflection for Disaggregated LLM Serving
  2. 2025DynaServe: Unified and Elastic Execution for Dynamic Disaggregated LLM Serving
  3. 2026StreamServe: Adaptive Speculative Flows for Low-Latency Disaggregated LLM Serving
  4. 2026DualScale: Energy-Efficient Disaggregated LLM Serving via Phase-Aware Placement and DVFS
  5. 2026SUN: Shared Use of Next-token Prediction for Efficient Multi-LLM Disaggregated Serving
  6. 2026LAPS: A Length-Aware-Prefill LLM Serving System
  7. 2026CoCoScale: Leveraging Layer-wise Scaling to Unlock the Potential of Online LLM Serving
  8. 2025BanaServe: Unified KV Cache and Dynamic Module Migration for Balancing Disaggregated LLM Serving in AI Infrastructure