# RouterBench: A Benchmark for Multi-LLM Routing System

arXiv: https://arxiv.org/abs/2403.12031 · HTML: https://mlsanity.com/paper/2403.12031/

## Key facts

- Cited by 238 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 54 references that are in the map.
- First posted on arXiv 18 Mar 2024.

## Builds on (most cited first, 30 of 39)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2205.10625/) (2022)
- [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://mlsanity.com/paper/2306.00978/) (2023)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [GLaM: Efficient Scaling of Language Models with Mixture-of-Experts](https://mlsanity.com/paper/2112.06905/) (2021)
- [FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance](https://mlsanity.com/paper/2305.05176/) (2023)
- [Yi: Open Foundation Models by 01.AI](https://mlsanity.com/paper/2403.04652/) (2024)
- [LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion](https://mlsanity.com/paper/2306.02561/) (2023)
- [Large Language Models Are Not Robust Multiple Choice Selectors](https://mlsanity.com/paper/2309.03882/) (2023)
- [SqueezeLLM: Dense-and-Sparse Quantization](https://mlsanity.com/paper/2306.07629/) (2023)
- [Learning Factored Representations in a Deep Mixture of Experts](https://mlsanity.com/paper/1312.4314/) (2013)
- [Large Language Model Routing with Benchmark Datasets](https://mlsanity.com/paper/2309.15789/) (2023)
- [Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Models](https://mlsanity.com/paper/2311.08692/) (2023)

## Built on by (most cited first, 30 of 238)

- [RouteLLM: Learning to Route LLMs with Preference Data](https://mlsanity.com/paper/2406.18665/) (2024)
- [RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Models](https://mlsanity.com/paper/2409.19886/) (2024)
- [Universal Model Routing for Efficient LLM Inference](https://mlsanity.com/paper/2502.08773/) (2025)
- [Harnessing Multiple Large Language Models: A Survey on LLM Ensemble](https://mlsanity.com/paper/2502.18036/) (2025)
- [A Unified Approach to Routing and Cascading for LLMs](https://mlsanity.com/paper/2410.10347/) (2024)
- [ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2503.09501/) (2025)
- [Router-R1: Teaching LLMs Multi-Round Routing and Aggregation via Reinforcement Learning](https://mlsanity.com/paper/2506.09033/) (2025)
- [Learning to Route LLMs with Confidence Tokens](https://mlsanity.com/paper/2410.13284/) (2024)
- [TensorOpera Router: A Multi-Model Router for Efficient LLM Inference](https://mlsanity.com/paper/2408.12320/) (2024)
- [EvoFlow: Evolving Diverse Agentic Workflows On The Fly](https://mlsanity.com/paper/2502.07373/) (2025)
- [EmbedLLM: Learning Compact Representations of Large Language Models](https://mlsanity.com/paper/2410.02223/) (2024)
- [MetaLLM: A High-performant and Cost-efficient Dynamic Framework for Wrapping LLMs](https://mlsanity.com/paper/2407.10834/) (2024)
- [What is the Role of Small Models in the LLM Era: A Survey](https://mlsanity.com/paper/2409.06857/) (2024)
- [CARROT: A Cost Aware Rate Optimal Router](https://mlsanity.com/paper/2502.03261/) (2025)
- [The Avengers: A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary Giants](https://mlsanity.com/paper/2505.19797/) (2025)
- [Don't Overthink It: A Survey of Efficient R1-style Large Reasoning Models](https://mlsanity.com/paper/2508.02120/) (2025)
- [Optimizing Model Selection for Compound AI Systems](https://mlsanity.com/paper/2502.14815/) (2025)
- [Eagle: Efficient Training-Free Router for Multi-LLM Inference](https://mlsanity.com/paper/2409.15518/) (2024)
- [Beyond GPT-5: Making LLMs Cheaper and Better via Performance-Efficiency Optimized Routing](https://mlsanity.com/paper/2508.12631/) (2025)
- [Smoothie: Label Free Language Model Routing](https://mlsanity.com/paper/2412.04692/) (2024)
- [Towards AI-Native Software Engineering (SE 3.0): A Vision and a Challenge Roadmap](https://mlsanity.com/paper/2410.06107/) (2024)
- [Causal LLM Routing: End-to-End Regret Minimization from Observational Data](https://mlsanity.com/paper/2505.16037/) (2025)
- [Capability Instruction Tuning: A New Paradigm for Dynamic LLM Routing](https://mlsanity.com/paper/2502.17282/) (2025)
- [Heterogeneous Swarms: Jointly Optimizing Model Roles and Weights for Multi-LLM Systems](https://mlsanity.com/paper/2502.04510/) (2025)
- [Route to Reason: Adaptive Routing for LLM and Reasoning Strategy Selection](https://mlsanity.com/paper/2505.19435/) (2025)
- [SkillOrchestra: Learning to Route Agents via Skill Transfer](https://mlsanity.com/paper/2602.19672/) (2026)
- [RouterArena: An Open Platform for Comprehensive Comparison of LLM Routers](https://mlsanity.com/paper/2510.00202/) (2025)
- [Doing More with Less: A Survey on Routing Strategies for Resource Optimisation in Large Language Model-Based Systems](https://mlsanity.com/paper/2502.00409/) (2025)
- [Do We Truly Need So Many Samples? Multi-LLM Repeated Sampling Efficiently Scales Test-Time Compute](https://mlsanity.com/paper/2504.00762/) (2025)
- [Dynamic Model Routing and Cascading for Efficient LLM Inference: A Survey](https://mlsanity.com/paper/2603.04445/) (2026)

## Abstract (arXiv)

As the range of applications for Large Language Models (LLMs) continues to grow, the demand for effective serving solutions becomes increasingly critical. Despite the versatility of LLMs, no single model can optimally address all tasks and applications, particularly when balancing performance with cost. This limitation has led to the development of LLM routing systems, which combine the strengths of various models to overcome the constraints of individual LLMs. Yet, the absence of a standardized benchmark for evaluating the performance of LLM routers hinders progress in this area. To bridge this gap, we present RouterBench, a novel evaluation framework designed to systematically assess the efficacy of LLM routing systems, along with a comprehensive dataset comprising over 405k inference outcomes from representative LLMs to support the development of routing strategies. We further propose a theoretical framework for LLM routing, and deliver a comparative analysis of various routing approaches through RouterBench, highlighting their potentials and limitations within our evaluation framework. This work not only formalizes and advances the development of LLM routing systems but also sets a standard for their assessment, paving the way for more accessible and economically viable LLM deployments. The code and data are available at https://github.com/withmartian/routerbench.