mlsanityData to 7 Oct 2026

RouteBalance: Fused Model Routing and Load Balancing for Heterogeneous LLM Serving

arXiv 2606.17949 ↗First posted 16 Jun 2026cs.DCcited by 1

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Heterogeneous LLM serving stacks split scheduling into two layers that optimize in isolation: model routers pick a model from quality and cost signals while ignoring instance load, and serving load balancers optimize queues while ignoring quality. We present RouteBalance, a serving-aware scheduling layer that fuses both into a single online assignment over concrete model instances, jointly trading off quality, latency, and cost. A batched in-process predictor stack and dead-reckoned instance state keep the joint decision cheap on the request hot path (≈32 ms at 12 req/s). On a 13-instance, 28-GPU heterogeneous cluster serving four model sizes, a single deployed RouteBalance stack traces the upper region of the three-way quality-cost-throughput frontier. Sweeping one weight vector reaches both the highest routing-decision quality (DeepEval 0.419, +0.013 over the strongest baseline, 95% CI [+0.005,+0.022]; the ordering holds when a second judge re-scores the actually served text) and, at its cost-priority corner, per-request cost that ties the cheapest baseline. With router engineering equalized against concurrent-scoring baseline variants we build, its balanced preset serves at 2.8 s and 30 req/s, leading 2.6 to 4.1× ahead of enhanced BEST-Route at high load. (Deploying those routers as published, one serial scoring call per request, makes them collapse 23× under load, a deployment-architecture effect we isolate separately, not the routing result.) A four-arm isolation shows the benefit follows from pricing latency at model-selection time; the learned predictors contribute calibration and SLO headroom rather than the headline frontier. Code: https://github.com/AKafakA/route-balance

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 39 of its 59 references that are in the map. First posted on arXiv 16 Jun 2026.

Builds on

39 of 59 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025Ranking Before Serving: Low-Latency LLM Serving via Pairwise Learning-to-Rank25 Sep
  2. 2025Efficient Training-Free Online Routing for High-Volume Multi-LLM Serving2 Sep
  3. 2025Astrolabe: Balancing Load in LLM Serving with Randomized Prediction-Guided Scheduling5 Aug
  4. 2025BucketServe: Bucket-Based Dynamic Batching for Smart and Efficient LLM Inference Serving23 Jul
  5. 2025ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environments13 Feb
  6. 2025Universal Model Routing for Efficient LLM Inference12 Feb
  7. 2025Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs2 Feb
  8. 2025HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-location15 Jan
  9. 2024LLMServingSim: A HW/SW Co-Simulation Infrastructure for LLM Inference Serving at Scale10 Aug
  10. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving24 Jun
  11. 2024Queue management for slo-oriented large language model serving5 Jun
  12. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving5 Jun
1 more
  1. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline22 May

Other references · 26

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
  4. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  5. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  6. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  7. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 633
  8. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  9. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  10. 2024The Faiss librarycited by 370
  11. 2025Small Language Models are the Future of Agentic AIcited by 254
  12. 2024RouterBench: A Benchmark for Multi-LLM Routing Systemcited by 222
14 more
  1. 2017The Case for Learned Index Structurescited by 188
  2. 2024Vidur: A Large-Scale Simulation Framework For LLM Inferencecited by 115
  3. 2025BEST-Route: Adaptive LLM Routing with Test-Time Optimal Computecited by 63
  4. 2024CodeUltraFeedback: An LLM-as-a-Judge Dataset for Aligning Large Language Models to Coding Preferencescited by 56
  5. 2023HexGen: Generative Inference of Large Language Model over Heterogeneous Environmentcited by 31
  6. 2025Adaptive LLM Routing under Budget Constraintscited by 21
  7. 2025HexGen-2: Disaggregated Generative Inference of LLMs in Heterogeneous Environmentcited by 20
  8. 2025When to Reason: Semantic Router for vLLMcited by 12
  9. 2025LLM Query Scheduling with Prefix Reuse and Latency Constraintscited by 8
  10. 2025Dynamic Quality-Latency Aware Routing for LLM Inference in Wireless Edge-Device Networkscited by 4
  11. 2025TimeBill: Time-Budgeted Inference for Large Language Modelscited by 4
  12. 2025Eva: Cost-Efficient Cloud-Based Cluster Schedulingcited by 3
  13. 2025Staggered Batch Scheduling: Co-optimizing Time-to-First-Token and Throughput for High-Efficiency LLM Inferencecited by 2
  14. 2026PARD: Enhancing Goodput for Inference Pipeline via Proactive Request Droppingcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026DEALS: Decentralized Expertise-Aware Load Serving for Multi-Agent LLM Systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Tackling the Data-Parallel Load Balancing Bottleneck in LLM Serving: Practical Online Routing at Scale
  2. 2026RouterWise: Joint Resource Allocation and Routing for Latency-Aware Multi-Model LLM Serving
  3. 2025A Predictive and Synergistic Two-Layer Scheduling Framework for LLM Serving
  4. 2026Cross-Model Autoscaling for Shared LLM Serving
  5. 2025FREESH: Fair, Resource- and Energy-Efficient Scheduling for LLM Serving on Heterogeneous GPUs
  6. 2026DualMap: Enabling Both Cache Affinity and Load Balancing for Distributed LLM Serving
  7. 2026Nitsum: Serving Tiered LLM Requests with Adaptive Tensor Parallelism
  8. 2026BOute: Cost-Efficient LLM Serving with Heterogeneous LLMs and GPUs via Multi-Objective Bayesian Optimization