mlsanityData to 14 Oct 2026

Calibrate, Then Route: A Measured Study of Learned Request Routing for Disaggregated LLM Serving

arXiv 2609.16206 ↗First posted 14 Sep 2026cs.AIcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Disaggregated LLM serving places compute heavy prefill and memory heavy decode on separate GPU pools. Systems such as DistServe, Splitwise, and Mooncake make this separation fast, but routing still determines which instances handle each request. We study a router that estimates the additional completion time on each instance using exact prompt length, predicted output length, post admission KV cache pressure, and SLO class. We develop the policy in a discrete event simulator and validate it on eight NVIDIA A40 GPUs, each running a vLLM engine, with NIXL transferring KV caches between pools. All workloads run at measured saturation. Across three mixed, bursty arrival traces, the calibrated router achieves the highest mean goodput at 0.864, compared with 0.835 to 0.847 for round robin, least loaded, and a length heuristic. It also shows the lowest variance across traces. It beats round robin and the length heuristic on all three traces and least loaded on two. On the third, it trails by 0.003, within run to run noise. Hardware calibration matters: simulator derived constants cost 4.5 goodput points and roughly 40 percent of the tail latency advantage, reducing the scorer to little more than queue counting. Benefits grow with decode pool size and traffic heterogeneity but disappear in pools with three instances, where queue counts are often enough. Under extreme scarcity, greedy cost minimization concentrates requests on the cheapest scored instance, and blind spreading performs better. With calibrated costs, the learned router matches the goodput of round robin using six GPUs instead of seven.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 18 of its 21 references that are in the map. First posted on arXiv 14 Sep 2026.

Builds on

18 of 21 references.

Closest ancestors · 7

Same direction, newest first

  1. 2026LAPS: A Length-Aware-Prefill LLM Serving System4 Jan
  2. 2024Efficient LLM Scheduling by Learning to Rank28 Aug
  3. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving24 Jun
  4. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving5 Jun
  5. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  6. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  7. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline22 May

Other references · 11

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  4. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  5. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  6. 2019MLPerf Inference Benchmarkcited by 199
  7. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 121
  8. 2024Vidur: A Large-Scale Simulation Framework For LLM Inferencecited by 117
  9. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 112
  10. 2024LLM Inference Serving: Survey of Recent Advances and Opportunitiescited by 46
  11. 2023AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Servingcited by 31

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026FleetOpt: Analytical Fleet Provisioning for LLM Inference with Compress-and-Route as Implementation Mechanism
  2. 2026CacheRoute: Planned Prefix-Affinity Routing for Large-Scale LLM Serving
  3. 2026RequestRouter: Request-Boundary Routing for Efficient Single-GPU LLM Inference
  4. 2026RouterWise: Joint Resource Allocation and Routing for Latency-Aware Multi-Model LLM Serving
  5. 2026Towards Load-Aware Prefill Deflection for Disaggregated LLM Serving
  6. 2026Ready Cohorts: Bounding GPU Opportunity and Avoiding Host Round Trips in LLM-Agent Control
  7. 2026Nitsum: Serving Tiered LLM Requests with Adaptive Tensor Parallelism
  8. 2026StreamServe: Adaptive Speculative Flows for Low-Latency Disaggregated LLM Serving