mlsanityData to 7 Oct 2026

TensorOpera Router: A Multi-Model Router for Efficient LLM Inference

arXiv 2408.12320 ↗First posted 22 Aug 2024cs.AI cs.LGcited by 59

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

With the rapid growth of Large Language Models (LLMs) across various domains, numerous new LLMs have emerged, each possessing domain-specific expertise. This proliferation has highlighted the need for quick, high-quality, and cost-effective LLM query response methods. Yet, no single LLM exists to efficiently balance this trilemma. Some models are powerful but extremely costly, while others are fast and inexpensive but qualitatively inferior. To address this challenge, we present TO-Router, a non-monolithic LLM querying system that seamlessly integrates various LLM experts into a single query interface and dynamically routes incoming queries to the most high-performant expert based on query's requirements. Through extensive experiments, we demonstrate that when compared to standalone expert models, TO-Router improves query efficiency by up to 40%, and leads to significant cost reductions of up to 30%, while maintaining or enhancing model performance by up to 10%.

Key facts. Cited by 59 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 21 of its 26 references that are in the map. First posted on arXiv 22 Aug 2024.

Builds on

21 of 26 references.

References · 21

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2021Program Synthesis with Large Language Modelscited by 3,509
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  9. 2023Mistral 7Bcited by 3,065
  10. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  11. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  12. 2024Mixtral of Expertscited by 1,785
9 more
  1. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  2. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  3. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  4. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 633
  5. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 354
  6. 2024BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domainscited by 299
  7. 2024RouterBench: A Benchmark for Multi-LLM Routing Systemcited by 222
  8. 2023Large Language Model Routing with Benchmark Datasetscited by 166
  9. 2024Harnessing the Power of Multiple Minds: Lessons Learned from LLM Routingcited by 43

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 31; 2026: 28.

59 papers.

Sort:

By area:

  1. 2025Harnessing Multiple Large Language Models: A Survey on LLM Ensemblecited by 86
  2. 2025Universal Model Routing for Efficient LLM Inferencecited by 80
  3. 2025MasRouter: Learning to Route LLMs for Multi-Agent Systemscited by 80
  4. 2025ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learningcited by 78
  5. 2025Router-R1: Teaching LLMs Multi-Round Routing and Aggregation via Reinforcement Learningcited by 64
  6. 2025Optimizing Model Selection for Compound AI Systemscited by 34
  7. 2025CITER: Collaborative Inference for Efficient Large Language Model Decoding with Token-Level Routingcited by 29
  8. 2025When One LLM Drools, Multi-LLM Collaboration Rulescited by 25
  9. 2026SkillOrchestra: Learning to Route Agents via Skill Transfercited by 22
  10. 2025Leveraging Uncertainty Estimation for Efficient LLM Routingcited by 21
  11. 2025Adaptive LLM Routing under Budget Constraintscited by 21
  12. 2025Doing More with Less: A Survey on Routing Strategies for Resource Optimisation in Large Language Model-Based Systemscited by 18
38 more
  1. 2025Cost-Aware Contrastive Routing for LLMscited by 16
  2. 2025xRouter: Training Cost-Aware LLMs Orchestration System via Reinforcement Learningcited by 16
  3. 2025Efficient Training-Free Online Routing for High-Volume Multi-LLM Servingcited by 13
  4. 2026R2-Router: A New Paradigm for LLM Routing with Reasoningcited by 13
  5. 2025FusionFactory: Fusing LLM Capabilities with Multi-LLM Log Datacited by 10
  6. 2026When Routing Collapses: On the Degenerate Convergence of LLM Routerscited by 10
  7. 2025MESS+: Dynamically Learned Inference-Time LLM Routing in Model Zoos with Service Level Guaranteescited by 8
  8. 2025RAGRouter: Learning to Route Queries to Multiple Retrieval-Augmented Language Modelscited by 8
  9. 2025Rethinking Predictive Modeling for LLM Routing: When Simple kNN Beats Complex Learned Routerscited by 7
  10. 2025Fast or Better? Balancing Accuracy and Cost in Retrieval-Augmented Generation with Flexible User Controlcited by 6
  11. 2025Toward Super Agent System with Hybrid AI Routerscited by 6
  12. 2025SATER: A Self-Aware and Token-Efficient Approach to Routing and Cascadingcited by 6
  13. 2025VL-RouterBench: A Benchmark for Vision-Language Model Routingcited by 6
  14. 2025IPR: Intelligent Prompt Routing with User-Controlled Quality-Cost Trade-offscited by 5
  15. 2025ProxRouter: Proximity-Weighted LLM Query Routing for Improved Robustness to Outlierscited by 5
  16. 2025One Head, Many Models: Cross-Attention Routing for Cost-Aware LLM Selectioncited by 4
  17. 2025Reference-Free Rating of LLM Responses via Latent Informationcited by 4
  18. 2025ECVL-ROUTER: Scenario-Aware Routing for Vision-Language Modelscited by 4
  19. 2026LAPS: A Length-Aware-Prefill LLM Serving Systemcited by 4
  20. 2026TwinRouterBench: Fast Static and Live Dynamic Evaluation for Realistic Agentic LLM Routingcited by 4
  21. 2025Near-Optimal Online Deployment and Routing for Streaming LLMscited by 3
  22. 2026HAPS: Hierarchical LLM Routing with Joint Architecture and Parameter Searchcited by 3
  23. 2026Federate the Router: Learning Language Model Routers with Sparse and Decentralized Evaluationscited by 3
  24. 2026Prompt-Dependent Ranking of Large Language Models with Uncertainty Quantificationcited by 3
  25. 2026Switchcraft: AI Model Router for Agentic Tool Callingcited by 3
  26. 2026From Sampled Outcomes to Capability Distributions: Rethinking Supervision for LLM Routingcited by 3
  27. 2026MentorCollab: Large-to-Small Inference-Time Mentorship for Concise Reasoning in Language Modelscited by 2
  28. 2026Picking the Right Specialist: Attentive Neural Process-based Selection of Task-Specialized Models as Tools for Agentic Healthcare Systemscited by 2
  29. 2025Meta-Router: Bridging Gold-standard and Preference-based Evaluations in Large Language Model Routingcited by 1
  30. 2026Privacy-Preserving LLMs Routingcited by 1
  31. 2026RouteJudge: An Open Platform for Reproducible and Preference-Aware LLM Routingcited by 1
  32. 2026Cost-Optimal LLM Routing with Limited User Feedback under User Satisfaction Guaranteescited by 1
  33. 2026LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routerscited by 1
  34. 2026FlexComp: One Model for Every Ratio in Context Compressioncited by 1
  35. 2026Carbon-Aware Routing for Function Calling in Edge-Cloud LLM Systemscited by 1
  36. 2025ECORE: Energy-Conscious Optimized Routing for Deep Learning Models at the Edgecited by 0
  37. 2025CoE-Ops: Collaboration of LLM-based Experts for AIOps Question-Answeringcited by 0
  38. 2026CR^2: Cost-Aware Risk-Controlled Routing for Wireless Device-Edge LLM Inferencecited by 0

50 most cited of 59 All 59 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniques
  2. 2026Dynamic Model Routing and Cascading for Efficient LLM Inference: A Survey
  3. 2025Lookahead Routing for Large Language Models
  4. 2025When to Reason: Semantic Router for vLLM
  5. 2026Measured Joules, Learned Routes: Learning to Route for Energy-Efficient LLM Serving
  6. 2024Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design
  7. 2024LLM Inference Unveiled: Survey and Roofline Model Insights
  8. 2025RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMs