mlsanityData to 7 Oct 2026

RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMs

arXiv 2503.10657 ↗First posted 8 Mar 2025cs.CL cs.AIcited by 58

Not on the map. Why?

Abstract (arXiv)

Routing large language models (LLMs) is a new paradigm that uses a router to recommend the best LLM from a pool of candidates for a given input. In this paper, our comprehensive analysis with more than 8,500 LLMs reveals a novel model-level scaling up phenomenon in Routing LLMs, i.e., a capable router can significantly enhance the performance of this paradigm as the number of candidates increases. This improvement can even surpass the performance of the best single model in the pool and many existing strong LLMs, confirming it a highly promising paradigm. However, the lack of comprehensive and open-source benchmarks for Routing LLMs has hindered the development of routers. In this paper, we introduce RouterEval, a benchmark tailored for router research, which includes over 200,000,000 performance records for 12 popular LLM evaluations across various areas such as commonsense reasoning, semantic understanding, etc., based on over 8,500 various LLMs. Using RouterEval, extensive evaluations of existing Routing LLM methods reveal that most still have significant room for improvement. See https://github.com/MilkThink-Lab/RouterEval for all data, code and tutorial.

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). Builds on 113 of its 163 references that are in the map. First posted on arXiv 8 Mar 2025.

Builds on

113 of 163 references.

References · 113

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  10. 2017Squeeze-and-Excitation Networkscited by 4,763
  11. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  12. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
38 more
  1. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  3. 2020Longformer: The Long-Document Transformercited by 3,362
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  5. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  6. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  7. 2016Matching Networks for One Shot Learningcited by 2,246
  8. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  9. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  10. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  11. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  12. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  13. 2018Graph Convolutional Neural Networks for Web-Scale Recommender Systemscited by 1,133
  14. 2017Neural Collaborative Filteringcited by 1,108
  15. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  16. 2019BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformercited by 952
  17. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  18. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  19. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  20. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  21. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 633
  22. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  23. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  24. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
  25. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  26. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  27. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
  28. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  29. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  30. 2015Deep Speech 2: End-to-End Speech Recognition in English and Mandarincited by 449
  31. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  32. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  33. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 354
  34. 2020Rethinking Pre-training and Self-trainingcited by 352
  35. 2023How Good Are GPT Models at Machine Translation? A Comprehensive Evaluationcited by 344
  36. 2021Scaling Laws for Transfercited by 265
  37. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  38. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247

50 most cited of 113 All 113 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 17; 2026: 41.

58 papers.

Sort:

By area:

  1. 2025The Avengers: A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary Giantscited by 37
  2. 2026LLMRouterBench: A Massive Benchmark and Unified Framework for LLM Routingcited by 35
  3. 2025Beyond GPT-5: Making LLMs Cheaper and Better via Performance-Efficiency Optimized Routingcited by 32
  4. 2025Internet of Agents: Fundamentals, Applications, and Challengescited by 31
  5. 2025RouterArena: An Open Platform for Comprehensive Comparison of LLM Routerscited by 19
  6. 2026Dynamic Model Routing and Cascading for Efficient LLM Inference: A Surveycited by 16
  7. 2025Self-Resource Allocation in Multi-Agent LLM Systemscited by 13
  8. 2025Intelligence per Watt: Measuring Intelligence Efficiency of Local AIcited by 13
  9. 2026MMR-Bench: A Comprehensive Benchmark for Multimodal LLM Routingcited by 13
  10. 2026R2-Router: A New Paradigm for LLM Routing with Reasoningcited by 13
  11. 2026When Routing Collapses: On the Degenerate Convergence of LLM Routerscited by 10
  12. 2026SWE-Router: Routing in Multi-turn Agentic Software Engineering Taskscited by 9
38 more
  1. 2026Budget-Aware Agentic Routing via Boundary-Guided Trainingcited by 8
  2. 2026Neural Chain-of-Thought Search: Searching the Optimal Reasoning Path to Enhance Large Language Modelscited by 7
  3. 2026The Replay Gap: Static Evaluation of Model Switching in LLM Agents Scores the Wrong Worldcited by 7
  4. 2025VL-RouterBench: A Benchmark for Vision-Language Model Routingcited by 6
  5. 2025Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniquescited by 5
  6. 2025IPR: Intelligent Prompt Routing with User-Controlled Quality-Cost Trade-offscited by 5
  7. 2025ProxRouter: Proximity-Weighted LLM Query Routing for Improved Robustness to Outlierscited by 5
  8. 2026PROTEUS: SLA-Aware Routing via Lagrangian RL for Multi-LLM Serving Systemscited by 4
  9. 2026RACER: Risk-Aware Calibrated Efficient Routing for Large Language Modelscited by 4
  10. 2026Route to Rome Attack: Directing LLM Routers to Expensive Models via Adversarial Suffix Optimizationcited by 4
  11. 2026HAPS: Hierarchical LLM Routing with Joint Architecture and Parameter Searchcited by 3
  12. 2026Federate the Router: Learning Language Model Routers with Sparse and Decentralized Evaluationscited by 3
  13. 2026AgentSelect: Benchmark for Narrative Query-to-Agent Recommendationcited by 3
  14. 2026Judge, Then Drive: A Critic-Centric Vision Language Action Framework for Autonomous Drivingcited by 3
  15. 2026From Sampled Outcomes to Capability Distributions: Rethinking Supervision for LLM Routingcited by 3
  16. 2026What Does a Routing Oracle Measure Under Stochastic Decoding? Coupling, Scorer Choice, and Single-Commit Ceilingscited by 3
  17. 2025Guarded Query Routing for Large Language Modelscited by 2
  18. 2026Minimizing Mismatch Risk: A Prototype-Based Routing Framework for Zero-shot LLM-generated Text Detectioncited by 2
  19. 2026Towards Fair and Comprehensive Evaluation of Routers in Collaborative LLM Systemscited by 2
  20. 2026Task-Aware Delegation Cues for LLM Agentscited by 2
  21. 2026Tandem: Riding Together with Large and Small Language Models for Efficient Reasoningcited by 2
  22. 2026SeqRoute: Global Budget-Aware Sequential LLM Routing via Offline Reinforcement Learningcited by 2
  23. 2026Most of the LLM Routing Gap Is Task Typecited by 2
  24. 2026Beyond Top-k Skill Retrieval: Diversity-Aware Skill Routing for LLM Agentscited by 2
  25. 2025LightRouter: Towards Efficient LLM Collaboration with Minimal Overheadcited by 1
  26. 2025R1-Ranker: Teaching LLM Rankers to Reasoncited by 1
  27. 2025DART: Dual Adaptive Refinement Transfer for Open-Vocabulary Multi-Label Recognitioncited by 1
  28. 2026ModelLens: Finding the Best for Your Task from Myriads of Modelscited by 1
  29. 2026BLUE: Toward Better Language Use in Efficient Vision-Language-Action Models for Autonomous Drivingcited by 1
  30. 2026RouteJudge: An Open Platform for Reproducible and Preference-Aware LLM Routingcited by 1
  31. 2026LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routerscited by 1
  32. 2025AssoCiAm: A Benchmark for Evaluating Association Thinking while Circumventing Ambiguitycited by 0
  33. 2025A Greedy PDE Router for Blending Neural Operators and Classical Methodscited by 0
  34. 2025RoBoN: Routed Online Best-of-n for Test-Time Scaling with Multiple LLMscited by 0
  35. 2026A self-evolving multi-role collaborative framework with fine-grained difficulty guidance for innovative mathematical problem generationcited by 0
  36. 2026Route-Induced Density and Stability (RIDE): Controlled Intervention and Mechanism Analysis of Routing-Style Meta Prompts on LLM Internal Statescited by 0
  37. 2026AgentOpt v0.1 Technical Report: Client-Side Optimization for LLM-Based Agentcited by 0
  38. 2026Resample or Reroute? Recoverable Stopping Debt Without Identified Action Selectioncited by 0

50 most cited of 58 All 58 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Rethinking Predictive Modeling for LLM Routing: When Simple kNN Beats Complex Learned Routers
  2. 2024TensorOpera Router: A Multi-Model Router for Efficient LLM Inference
  3. 2025Lookahead Routing for Large Language Models
  4. 2025When to Reason: Semantic Router for vLLM
  5. 2026Evaluating Small Language Models for Front-Door Routing: A Harmonized Benchmark and Synthetic-Traffic Experiment
  6. 2025Route-and-Reason: Scaling Large Language Model Reasoning with Reinforced Model Router
  7. 2025TravelBench : Exploring LLM Performance in Low-Resource Domains
  8. 2025Multilingual Routing in Mixture-of-Experts