mlsanityData to 14 Oct 2026

Efficient Training-Free Online Routing for High-Volume Multi-LLM Serving

arXiv 2509.02718 ↗First posted 2 Sep 2025cs.DB cs.AI cs.LGcited by 13

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Increasing demand for Large Language Models (LLMs) services imposes substantial deployment and computation costs on providers. LLM routing offers a cost-efficient solution by directing queries to the optimal LLM based on model and query features. However, existing works primarily focus on offline scenarios and struggle to adapt to online settings with high query volume and constrained token budgets. In this work, we introduce the first training-free algorithm for online routing scenarios. Our algorithm leverages approximate nearest neighbor search to efficiently estimate query features and performs a one-time optimization over a small set of initial queries to learn a routing strategy that guides future routing. We provide theoretical guarantees demonstrating that our algorithm achieves a competitive ratio of 1 - o(1) under natural assumptions, which is further validated by extensive experiments across 3 benchmark datasets and 8 baselines, showing an average improvement of 3.55× in overall performance, 1.85× in cost efficiency, and nearly 4.25× in throughput. Our code is available at https://github.com/fzwark/PORT.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 45 of its 74 references that are in the map. First posted on arXiv 2 Sep 2025.

Builds on

45 of 74 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Universal Model Routing for Efficient LLM Inference12 Feb
  2. 2024TensorOpera Router: A Multi-Model Router for Efficient LLM Inference22 Aug

Other references · 43

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2021Program Synthesis with Large Language Modelscited by 3,637
  9. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  10. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  11. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  12. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
31 more
  1. 2016Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphscited by 861
  2. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
  3. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 723
  4. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  5. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
  6. 2016CVXPY: A Python-Embedded Modeling Language for Convex Optimizationcited by 641
  7. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 578
  8. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  9. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 372
  10. 2023Mathematical Capabilities of ChatGPTcited by 268
  11. 2024RouterBench: A Benchmark for Multi-LLM Routing Systemcited by 238
  12. 2023MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoningcited by 201
  13. 2023Large Language Model Routing with Benchmark Datasetscited by 172
  14. 2024GraphRouter: A Graph-based Router for LLM Selectionscited by 165
  15. 2023Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Modelscited by 145
  16. 2023Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoningcited by 143
  17. 2023AutoMix: Automatically Mixing Language Modelscited by 125
  18. 2024RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systemscited by 117
  19. 2023Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modelingcited by 95
  20. 2015Parallelizing the dual revised simplex methodcited by 89
  21. 2024Scientific Large Language Models: A Survey on Biological & Chemical Domainscited by 80
  22. 2025MixLLM: Dynamic Routing in Mixed Large Language Modelscited by 59
  23. 2023Fusing Models with Complementary Expertisecited by 46
  24. 2025CARROT: A Cost Aware Rate Optimal Routercited by 39
  25. 2024Routoo: Learning to Route to Large Language Models Effectivelycited by 34
  26. 2023Tryage: Real-time, intelligent Routing of User Prompts to Large Language Modelscited by 30
  27. 2023OrchestraLLM: Efficient Orchestration of Language Models for Dialogue State Trackingcited by 19
  28. 2023AI Insights into Theoretical Physics and the Swampland Program: A Journey Through the Cosmos with ChatGPTcited by 10
  29. 2025Cerebrum (AIOS SDK): A Platform for Agent Development, Deployment, Distribution, and Discoverycited by 6
  30. 2017Online Submodular Welfare Maximization: Greedy Beats 1/2 in Random Ordercited by 4
  31. 2024A Bi-metric Framework for Fast Similarity Searchcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 13.

13 papers.

Sort:

By area:

  1. 2026AutoSOTA: An End-to-End Automated Research System for State-of-the-Art AI Model Discoverycited by 23
  2. 2026Dynamic Model Routing and Cascading for Efficient LLM Inference: A Surveycited by 19
  3. 2026GenRouter: Unified Workflow Routing for Agentic Image Generationcited by 5
  4. 2026Robust Batch-Level Query Routing for Large Language Models under Cost and Capacity Constraintscited by 3
  5. 2026Optimal Splitting of Language Models from Mixtures to Specialized Domainscited by 1
  6. 2026The Routing Plateau: Understanding the Accuracy Limits of LLM Routerscited by 1
  7. 2026RouteBalance: Fused Model Routing and Load Balancing for Heterogeneous LLM Servingcited by 1
  8. 2026The Capability Frontier: Benchmarks Miss 82% of Model Performancecited by 1
  9. 2026Training-Free versus Training-Based Intent Classification in LLMs: Accuracy, Robustness, and Failure Modescited by 1
  10. 2026Cost-Ordered Feasibility for Multi-Armed Bandits with Cost Subsidycited by 0
  11. 2026Capturing LLM Capabilities via Evidence-Calibrated Query Clusteringcited by 0
  12. 2026Online Linear Programming for Multi-Objective Routing in LLM Servingcited by 0
1 more
  1. 2026Unified AI Gateway: A Framework for Joint Model Routing and KV Cache Managementcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Adaptive LLM Routing under Budget Constraints
  2. 2025Doing More with Less: A Survey on Routing Strategies for Resource Optimisation in Large Language Model-Based Systems
  3. 2025OmniRouter: Budget and Performance Controllable Multi-LLM Routing
  4. 2025Rethinking Predictive Modeling for LLM Routing: When Simple kNN Beats Complex Learned Routers
  5. 2025LLM Bandit: Cost-Efficient LLM Generation via Preference-Conditioned Dynamic Routing
  6. 2024Ada-K Routing: Boosting the Efficiency of MoE-based LLMs
  7. 2025Dynamic LLM Routing and Selection based on User Preferences: Balancing Performance, Cost, and Ethics
  8. 2026TokenRouter: Efficient Serving System for Token-Level LLM Routing