mlsanityData to 14 Oct 2026

RouterWise: Joint Resource Allocation and Routing for Latency-Aware Multi-Model LLM Serving

arXiv 2604.10907 ↗First posted 13 Apr 2026cs.NI cs.DCcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Multi-model LLM routing has emerged as an effective approach for reducing serving cost and latency while maintaining output quality by assigning each prompt to an appropriate model. However, prior routing methods typically assume that each model has a fixed latency. In real deployments, this assumption is inaccurate: multiple models often share limited GPU resources, and a model's latency depends strongly on both its allocated resources and the request load induced by the routing policy. Consequently, routing and resource allocation are tightly coupled. In this work, we study joint resource allocation and routing for latency-aware multi-model LLM serving in GPU clusters. Given a set of deployed models and a latency service-level objective (SLO), we seek a system setup and routing policy that maximize overall output quality while satisfying the latency target. We formalize this problem as a constrained joint optimization over deployment setup and routing fractions, and propose RouterWise, which combines a dual-price formulation for score-maximizing routing with setup-specific latency models derived from system profiling. RouterWise searches over feasible system setups and, for each fixed setup, computes the best routing policy under the latency target. Our results show that even on the same GPU cluster, achievable output-quality score can vary by up to 87% across retained setups, highlighting that resource allocation is a key determinant of routing performance.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 11 of its 26 references that are in the map. First posted on arXiv 13 Apr 2026.

Builds on

11 of 26 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025gLLM: Global Balanced Pipeline Parallelism System for Distributed LLM Serving with Token Throttling21 Apr
  2. 2024MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving2 Apr

Other references · 9

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  3. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
  4. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 372
  5. 2024Learning to Route LLMs with Confidence Tokenscited by 67
  6. 2025BEST-Route: Adaptive LLM Routing with Test-Time Optimal Computecited by 67
  7. 2025Hetis: Serving LLMs in Heterogeneous GPU Clusters with Fine-grained and Dynamic Parallelismcited by 9
  8. 2014A Tutorial on Dual Decomposition and Lagrangian Relaxation for Inference in Natural Language Processingcited by 8
  9. 2026IntroLM: Introspective Language Models via Prefilling-Time Self-Evaluationcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026RouteBalance: Fused Model Routing and Load Balancing for Heterogeneous LLM Serving
  2. 2025Boosting LLM Serving through Spatial-Temporal GPU Resource Sharing
  3. 2026BOute: Cost-Efficient LLM Serving with Heterogeneous LLMs and GPUs via Multi-Objective Bayesian Optimization
  4. 2026Calibrate, Then Route: A Measured Study of Learned Request Routing for Disaggregated LLM Serving
  5. 2024ConServe: Fine-Grained GPU Harvesting for LLM Online and Offline Co-Serving
  6. 2026Data Driven Optimization of GPU efficiency for Distributed LLM-Adapter Serving
  7. 2026FleetOpt: Analytical Fleet Provisioning for LLM Inference with Compress-and-Route as Implementation Mechanism
  8. 2026LLMVisor: A Real-Time Latency Attribution Model for Multi-Tenant LLM Serving