mlsanityData to 7 Oct 2026

RouterBench: A Benchmark for Multi-LLM Routing System

arXiv 2403.12031 ↗First posted 18 Mar 2024cs.LG cs.AIcited by 222

Not on the map. Why?

Cited by 51 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

As the range of applications for Large Language Models (LLMs) continues to grow, the demand for effective serving solutions becomes increasingly critical. Despite the versatility of LLMs, no single model can optimally address all tasks and applications, particularly when balancing performance with cost. This limitation has led to the development of LLM routing systems, which combine the strengths of various models to overcome the constraints of individual LLMs. Yet, the absence of a standardized benchmark for evaluating the performance of LLM routers hinders progress in this area. To bridge this gap, we present RouterBench, a novel evaluation framework designed to systematically assess the efficacy of LLM routing systems, along with a comprehensive dataset comprising over 405k inference outcomes from representative LLMs to support the development of routing strategies. We further propose a theoretical framework for LLM routing, and deliver a comparative analysis of various routing approaches through RouterBench, highlighting their potentials and limitations within our evaluation framework. This work not only formalizes and advances the development of LLM routing systems but also sets a standard for their assessment, paving the way for more accessible and economically viable LLM deployments. The code and data are available at https://github.com/withmartian/routerbench.

Key facts. Cited by 222 papers in the map (data through 7 Oct 2026). Builds on 39 of its 54 references that are in the map. First posted on arXiv 18 Mar 2024.

Builds on

39 of 54 references.

References · 39

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  9. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  12. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
27 more
  1. 2021Program Synthesis with Large Language Modelscited by 3,509
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  3. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  5. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  6. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  7. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  8. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  9. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  10. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  11. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  12. 2024Yi: Open Foundation Models by 01.AIcited by 706
  13. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  14. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
  15. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  16. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 275
  17. 2023Large Language Model Routing with Benchmark Datasetscited by 166
  18. 2023Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Modelscited by 141
  19. 2023Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoningcited by 140
  20. 2024When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboardscited by 134
  21. 2023AutoMix: Automatically Mixing Language Modelscited by 116
  22. 2023Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modelingcited by 94
  23. 2023Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Modelscited by 65
  24. 2023Getting MoRE out of Mixture of Language Model Reasoning Expertscited by 44
  25. 2023Tryage: Real-time, intelligent Routing of User Prompts to Large Language Modelscited by 28
  26. 2023DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelinescited by 27
  27. 2024Blending Is All You Need: Cheaper, Better Alternative to Trillion-Parameters LLMcited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 21; 2025: 69; 2026: 132.

222 papers.

Sort:

By area:

  1. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 633
  2. 2025Harnessing Multiple Large Language Models: A Survey on LLM Ensemblecited by 86
  3. 2024RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Modelscited by 84
  4. 2025Universal Model Routing for Efficient LLM Inferencecited by 80
  5. 2025ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learningcited by 78
  6. 2024A Unified Approach to Routing and Cascading for LLMscited by 71
  7. 2024Learning to Route LLMs with Confidence Tokenscited by 66
  8. 2025Router-R1: Teaching LLMs Multi-Round Routing and Aggregation via Reinforcement Learningcited by 64
  9. 2024TensorOpera Router: A Multi-Model Router for Efficient LLM Inferencecited by 59
  10. 2025EvoFlow: Evolving Diverse Agentic Workflows On The Flycited by 49
  11. 2024EmbedLLM: Learning Compact Representations of Large Language Modelscited by 46
  12. 2024What is the Role of Small Models in the LLM Era: A Surveycited by 41
38 more
  1. 2024MetaLLM: A High-performant and Cost-efficient Dynamic Framework for Wrapping LLMscited by 39
  2. 2025The Avengers: A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary Giantscited by 37
  3. 2025Don't Overthink It: A Survey of Efficient R1-style Large Reasoning Modelscited by 36
  4. 2024Eagle: Efficient Training-Free Router for Multi-LLM Inferencecited by 34
  5. 2025CARROT: A Cost Aware Rate Optimal Routercited by 34
  6. 2025Optimizing Model Selection for Compound AI Systemscited by 34
  7. 2025Beyond GPT-5: Making LLMs Cheaper and Better via Performance-Efficiency Optimized Routingcited by 32
  8. 2024Smoothie: Label Free Language Model Routingcited by 31
  9. 2024Towards AI-Native Software Engineering (SE 3.0): A Vision and a Challenge Roadmapcited by 30
  10. 2025Causal LLM Routing: End-to-End Regret Minimization from Observational Datacited by 27
  11. 2025Capability Instruction Tuning: A New Paradigm for Dynamic LLM Routingcited by 25
  12. 2025Route to Reason: Adaptive Routing for LLM and Reasoning Strategy Selectioncited by 23
  13. 2025Heterogeneous Swarms: Jointly Optimizing Model Roles and Weights for Multi-LLM Systemscited by 22
  14. 2026SkillOrchestra: Learning to Route Agents via Skill Transfercited by 22
  15. 2025Do We Truly Need So Many Samples? Multi-LLM Repeated Sampling Efficiently Scales Test-Time Computecited by 20
  16. 2025RouterArena: An Open Platform for Comprehensive Comparison of LLM Routerscited by 19
  17. 2025Doing More with Less: A Survey on Routing Strategies for Resource Optimisation in Large Language Model-Based Systemscited by 18
  18. 2025Learning to Deliberate: Meta-policy Collaboration for Agentic LLMs with Multi-agent Reinforcement Learningcited by 18
  19. 2024METIS: Fast Quality-Aware RAG Systems with Configuration Adaptationcited by 16
  20. 2025Cost-Aware Contrastive Routing for LLMscited by 16
  21. 2026Dynamic Model Routing and Cascading for Efficient LLM Inference: A Surveycited by 16
  22. 2024Cascade-Aware Training of Language Modelscited by 15
  23. 2025Maestro: Learning to Collaborate via Conditional Listwise Policy Optimization for Multi-Agent LLMscited by 14
  24. 2025Efficient Training-Free Online Routing for High-Volume Multi-LLM Servingcited by 13
  25. 2025Learning to Route LLMs from Bandit Feedback: One Policy, Many Trade-offscited by 13
  26. 2025Intelligence per Watt: Measuring Intelligence Efficiency of Local AIcited by 13
  27. 2026MMR-Bench: A Comprehensive Benchmark for Multimodal LLM Routingcited by 13
  28. 2026R2-Router: A New Paradigm for LLM Routing with Reasoningcited by 13
  29. 2026EvolveRouter: Co-Evolving Routing and Prompt for Multi-Agent Question Answeringcited by 13
  30. 2025Mixture of Thoughts: Learning to Aggregate What Experts Think, Not Just What They Saycited by 12
  31. 2025Life-Cycle Routing Vulnerabilities of LLM Routercited by 10
  32. 2025Geo-OLM: Enabling Sustainable Earth Observation Studies with Cost-Efficient Open Language Models & State-Driven Workflowscited by 10
  33. 2025FusionFactory: Fusing LLM Capabilities with Multi-LLM Log Datacited by 10
  34. 2026When Routing Collapses: On the Degenerate Convergence of LLM Routerscited by 10
  35. 2026GraphPlanner: Graph Memory-Augmented Agentic Routing for Multi-Agent LLMscited by 10
  36. 2026When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Modelscited by 10
  37. 2025Arch-Router: Aligning LLM Routing with Human Preferencescited by 9
  38. 2026Exploring Knowledge Purification in Multi-Teacher Knowledge Distillation for LLMscited by 9

50 most cited of 222 All 222 (JSON)