mlsanityData to 7 Oct 2026

LLMRouterBench: A Massive Benchmark and Unified Framework for LLM Routing

arXiv 2601.07206 ↗First posted 12 Jan 2026cs.AIcited by 35

Not on the map. Why?

Abstract (arXiv)

Large language model (LLM) routing assigns each query to the most suitable model from an ensemble. We introduce LLMRouterBench, a large-scale benchmark and unified framework for LLM routing. It comprises over 400K instances from 21 datasets and 33 models. Moreover, it provides comprehensive metrics for both performance-oriented routing and performance-cost trade-off routing, and integrates 10 representative routing baselines. Using LLMRouterBench, we systematically re-evaluate the field. While confirming strong model complementarity-the central premise of LLM routing-we find that many routing methods exhibit similar performance under unified evaluation, and several recent approaches, including commercial routers, fail to reliably outperform a simple baseline. Meanwhile, a substantial gap remains to the Oracle, driven primarily by persistent model-recall failures. We further show that backbone embedding models have limited impact, that larger ensembles exhibit diminishing returns compared to careful model curation, and that the benchmark also enables latency-aware analysis. All code and data are available at https://github.com/ynulihao/LLMRouterBench.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Builds on 39 of its 62 references that are in the map. First posted on arXiv 12 Jan 2026.

Builds on

39 of 62 references.

References · 39

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2023Let's Verify Step by Stepcited by 3,598
  4. 2021Program Synthesis with Large Language Modelscited by 3,509
  5. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  6. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  7. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  8. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  9. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  10. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  11. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  12. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
27 more
  1. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  2. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  3. 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 636
  4. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 633
  5. 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 558
  6. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 552
  7. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  8. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  9. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 446
  10. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 354
  11. 2024GraphRouter: A Graph-based Router for LLM Selectionscited by 154
  12. 2024On Memorization of Large Language Models in Logical Reasoningcited by 114
  13. 2024MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmarkcited by 103
  14. 2025Universal Model Routing for Efficient LLM Inferencecited by 80
  15. 2024UltraMedical: Building Specialized Generalists in Biomedicinecited by 67
  16. 2024KOR-Bench: Benchmarking Language Models on Knowledge-Orthogonal Reasoning Taskscited by 67
  17. 2024Are Your LLMs Capable of Stable Reasoning?cited by 65
  18. 2025BEST-Route: Adaptive LLM Routing with Test-Time Optimal Computecited by 63
  19. 2025RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMscited by 58
  20. 2025Beyond GPT-5: Making LLMs Cheaper and Better via Performance-Efficiency Optimized Routingcited by 32
  21. 2025ICL-Router: In-Context Learned Model Representations for LLM Routingcited by 26
  22. 2025Capability Instruction Tuning: A New Paradigm for Dynamic LLM Routingcited by 25
  23. 2025RouterArena: An Open Platform for Comprehensive Comparison of LLM Routerscited by 19
  24. 2025Mixture of Thoughts: Learning to Aggregate What Experts Think, Not Just What They Saycited by 12
  25. 2025RadialRouter: Structured Representation for Efficient and Robust Large Language Models Routingcited by 11
  26. 2025ProxRouter: Proximity-Weighted LLM Query Routing for Improved Robustness to Outlierscited by 5
  27. 2025Towards Generalized Routing: Model and Agent Orchestration for Adaptive and Efficient Inferencecited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 35.

35 papers.

Sort:

By area:

  1. 2026Dynamic Model Routing and Cascading for Efficient LLM Inference: A Surveycited by 16
  2. 2026Agent-as-a-Router: Agentic Model Routing for Coding Taskscited by 10
  3. 2026REFLEX with Jev for Efficient Selective Control in LLM Agentscited by 9
  4. 2026TwinRouterBench: Fast Static and Live Dynamic Evaluation for Realistic Agentic LLM Routingcited by 4
  5. 2026Agentic Routing: The Harness-Native Data Flywheelcited by 4
  6. 2026Switchcraft: AI Model Router for Agentic Tool Callingcited by 3
  7. 2026What Does a Routing Oracle Measure Under Stochastic Decoding? Coupling, Scorer Choice, and Single-Commit Ceilingscited by 3
  8. 2026LLM Router: Rethinking Routing with Prefill Activationscited by 2
  9. 2026Uno-Orchestra: Parsimonious Agent Routing via Selective Delegationcited by 2
  10. 2026Most of the LLM Routing Gap Is Task Typecited by 2
  11. 2026The Workload-Router-Pool Architecture for LLM Inference Optimization: A Vision Paper from the vLLM Semantic Router Projectcited by 1
  12. 2026Latency-Quality Routing for Functionally Equivalent Tools in LLM Agentscited by 1
23 more
  1. 2026TRACE-ROUTER: Task-Consistent and Adaptive Online Routing for Agentic AIcited by 1
  2. 2026Signed Rescue Routing: Harm-Aware Cascades for Efficient LLM Inferencecited by 1
  3. 2026Control the Harness, Control the Cost: Routing and Governing AI Coding Agents in the Enterprisecited by 1
  4. 2026A Large-Scale Benchmark and Risk Assessment of Traffic Analysis Attacks on Cloud LLM Servicescited by 1
  5. 2026Model Routing as a Trust Problem: Route Receipts for Adaptive AI Systemscited by 0
  6. 2026The Routing Plateau: Understanding the Accuracy Limits of LLM Routerscited by 0
  7. 2026LLMoxie: Exploring Agentic AI for Scientific Software Developmentcited by 0
  8. 2026Aleena: Alignment Agent for Research Software Engineering Collaborationscited by 0
  9. 2026Resample or Reroute? Recoverable Stopping Debt Without Identified Action Selectioncited by 0
  10. 2026VDAR-Router: Adaptive LLMs Routing via Verbalized Query Difficulty Analysis Retrievalcited by 0
  11. 2026Detecting an Effect Is Not Learning to Act on It: A Reward-SNR Floor for LLM Acquisition Agentscited by 0
  12. 2026SCOPE-Router: Cost-Aware Open-Set VLM Routing for Execution-Oriented Taskscited by 0
  13. 2026RLCascadeRouter: Quality-Estimator-Free Cascade Routing via Reinforcement Learningcited by 0
  14. 2026Which LLM for Which Work? Budgeted Model Allocation under Uncertain Evaluationcited by 0
  15. 2026Mo' Models, Mo' Problems: How to best select model pools when designing Multi-Agent Systemscited by 0
  16. 2026Measured Joules, Learned Routes: Learning to Route for Energy-Efficient LLM Servingcited by 0
  17. 2026Canopy: Exploiting Piecewise Smooth Tree Priors for Multi-Fidelity Banditscited by 0
  18. 2026Oracle Gaps in Reliability Coverage: Sampling Noise or Policy Specialization?cited by 0
  19. 2026RSI-Router: Evolving Subtask-Level LLM Routing and Skills for Cost-Efficient Agentscited by 0
  20. 2026SeLMRoute: Probabilistic Semantic Evidence for Large Language Model Routingcited by 0
  21. 2026Pretrain Once, Route Anywhere: Towards a Foundation Model for LLM Routingcited by 0
  22. 2026Routing Should Pay for Itself: Sparse Supervision for Economical LLM Routingcited by 0
  23. 2026False Floors: LLM Safety Routing Evaluations Break Under Distribution Shiftcited by 0