mlsanityData to 7 Oct 2026

Token-Budget-Aware Pool Routing for Cost-Efficient LLM Inference

arXiv 2604.09613 ↗First posted 13 Mar 2026cs.DC cs.AIcited by 2

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Production vLLM fleets provision every instance for worst-case context length, wasting 4-8x concurrency on the 80-95% of requests that are short and simultaneously triggering KV-cache failures -- OOM crashes, preemption storms, and request rejections. Both problems share a single root cause: configuration-traffic mismatch. We propose token-budget-aware pool routing: estimate each request's total token budget using a self-calibrating per-category bytes-per-token ratio, then dispatch it to one of two vLLM pools -- a high-throughput short pool or a high-capacity long pool -- each right-sized for its workload class. The ratio is learned online via exponential moving average from usage.prompttokens feedback, requiring no tokenizer. A closed-form cost model, savings = alpha * (1 - 1/rho), predicts fleet-level GPU savings from two observable quantities: the short-traffic fraction alpha and the throughput gain ratio rho. On traces from the Azure LLM Inference Dataset and LMSYS-Chat-1M serving Llama-3-70B on A100 GPUs, token-budget routing reduces GPU instances by 17-39% (1.2-2.0M/yr at 1,000 req/s), with savings verified by a self-contained discrete-event simulator. A case study projecting Qwen3-235B-A22B on AMD MI300X at 10,000 req/s shows 15.4M/yr in savings. The algorithm adds O(1) dispatch overhead, self-calibrates across content types without a tokenizer, and composes with PagedAttention, continuous batching, and prefill-decode disaggregation.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 19 of its 27 references that are in the map. First posted on arXiv 13 Mar 2026.

Builds on

19 of 27 references.

Closest ancestors · 7

Same direction, newest first

  1. 2026OrbitFlow: SLO-Aware Long-Context LLM Serving with Fine-Grained KV Cache Reconfiguration5 Jan
  2. 2025ServeGen: Workload Characterization and Generation of Large Language Model Serving in Production15 May
  3. 2025Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs2 Feb
  4. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving24 Jun
  5. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving5 Jun
  6. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  7. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems31 Jan

Other references · 12

  1. 2025Qwen3 Technical Reportcited by 8,195
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  4. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  5. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  6. 2023Compressing Context to Enhance Inference Efficiency of Large Language Modelscited by 185
  7. 2024Vidur: A Large-Scale Simulation Framework For LLM Inferencecited by 115
  8. 2023AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Servingcited by 31
  9. 2025KVShare: An LLM Service System with Efficient and Effective Multi-Tenant KV Cache Reusecited by 22
  10. 2025LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Modelscited by 9
  11. 2026The Script Tax: Measuring Tokenization-Driven Efficiency and Latency Disparities in Multilingual Language Modelscited by 3
  12. 2026EWSJF: An Adaptive Scheduler with Hybrid Partitioning for Mixed-Workload LLM Inferencecited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026FleetOpt: Analytical Fleet Provisioning for LLM Inference with Compress-and-Route as Implementation Mechanismcited by 3
  2. 2026The Workload-Router-Pool Architecture for LLM Inference Optimization: A Vision Paper from the vLLM Semantic Router Projectcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026Dual-Pool Token-Budget Routing for Cost-Efficient and Reliable LLM Serving
  2. 2026Position: LLM Inference Should Be Evaluated as Energy-to-Token Production
  3. 2026TokenRouter: Efficient Serving System for Token-Level LLM Routing
  4. 2026Beyond Per-Token Pricing: A Concurrency-Aware Methodology for LLM Infrastructure Cost Estimation
  5. 2026Token Management in Multi-Tenant AI Inference Platforms
  6. 2026Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving
  7. 2026Position: LLM Serving Needs Mathematical Optimization and Algorithmic Foundations, Not Just Heuristics
  8. 2026Calibrate, Then Route: A Measured Study of Learned Request Routing for Disaggregated LLM Serving