mlsanityData to 14 Oct 2026

Large Language Model Routing with Benchmark Datasets

arXiv 2309.15789 ↗First posted 27 Sep 2023cs.CL cs.LGcited by 172

Not on the map. Why?

Cited by 24 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

There is a rapidly growing number of open-source Large Language Models (LLMs) and benchmark datasets to compare them. While some models dominate these benchmarks, no single model typically achieves the best accuracy in all tasks and use cases. In this work, we address the challenge of selecting the best LLM out of a collection of models for new tasks. We propose a new formulation for the problem, in which benchmark datasets are repurposed to learn a "router" model for this LLM selection, and we show that this problem can be reduced to a collection of binary classification tasks. We demonstrate the utility and limitations of learning model routers from various benchmark datasets, where we consistently improve performance upon using any single model for all tasks.

Key facts. Cited by 172 papers in the map (data through 14 Oct 2026). Builds on 38 of its 56 references that are in the map. First posted on arXiv 27 Sep 2023.

Builds on

38 of 56 references.

References · 38

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  6. 2017On Calibration of Modern Neural Networkscited by 4,960
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  10. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  11. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  12. 2019Invariant Risk Minimizationcited by 1,854
26 more
  1. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  2. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,375
  3. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
  4. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  5. 2016Deep CORAL: Correlation Alignment for Deep Domain Adaptationcited by 1,200
  6. 2020In Search of Lost Domain Generalizationcited by 1,054
  7. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
  8. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  9. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  10. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 422
  11. 2023GPTScore: Evaluate as You Desirecited by 361
  12. 2019Scaling and Benchmarking Self-Supervised Visual Representation Learningcited by 201
  13. 2018Model Evaluation, Model Selection, and Algorithm Selection in Machine Learningcited by 157
  14. 2022Leveraging Unlabeled Data to Predict Out-of-Distribution Performancecited by 145
  15. 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141
  16. 2021SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarizationcited by 132
  17. 2021Assessing Generalization of SGD via Disagreementcited by 117
  18. 2021Predicting with Confidence on Unseen Distributionscited by 116
  19. 2022SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarizationcited by 61
  20. 2021Mandoline: Model Evaluation under Distribution Shiftcited by 55
  21. 2021Detecting Errors and Estimating Accuracy on Unlabeled Data with Self-training Ensemblescited by 54
  22. 2022Predicting Out-of-Distribution Error with the Projection Normcited by 50
  23. 2020FrugalML: How to Use ML Prediction APIs More Accurately and Cheaplycited by 33
  24. 2022Understanding new tasks through the lens of training data via exponential tiltingcited by 11
  25. 2022Predicting Out-of-Domain Generalization with Neighborhood Invariancecited by 4
  26. 2022Estimation of prediction error with known covariate shiftcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 38; 2025: 61; 2026: 71.

172 papers.

Sort:

By area:

  1. 2024Mixture-of-Agents Enhances Large Language Model Capabilitiescited by 438
  2. 2024RouterBench: A Benchmark for Multi-LLM Routing Systemcited by 238
  3. 2024Chain of Agents: Large Language Models Collaborating on Long-Context Taskscited by 206
  4. 2024More Agents Is All You Needcited by 180
  5. 2023Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Modelscited by 145
  6. 2024Language Model Cascades: Token-level uncertainty and beyondcited by 121
  7. 2024RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Modelscited by 88
  8. 2025Universal Model Routing for Efficient LLM Inferencecited by 86
  9. 2025Harnessing Multiple Large Language Models: A Survey on LLM Ensemblecited by 86
  10. 2024A Unified Approach to Routing and Cascading for LLMscited by 80
  11. 2024Towards Modular LLMs by Building and Reusing a Library of LoRAscited by 72
  12. 2024Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel Collaborationcited by 67
38 more
  1. 2025BEST-Route: Adaptive LLM Routing with Test-Time Optimal Computecited by 67
  2. 2024Learning to Route Among Specialized Experts for Zero-Shot Generalizationcited by 65
  3. 2025RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMscited by 61
  4. 2024TensorOpera Router: A Multi-Model Router for Efficient LLM Inferencecited by 60
  5. 2025MixLLM: Dynamic Routing in Mixed Large Language Modelscited by 59
  6. 2024A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learningcited by 52
  7. 2024EmbedLLM: Learning Compact Representations of Large Language Modelscited by 48
  8. 2024Harnessing the Power of Multiple Minds: Lessons Learned from LLM Routingcited by 46
  9. 2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 44
  10. 2024MetaLLM: A High-performant and Cost-efficient Dynamic Framework for Wrapping LLMscited by 41
  11. 2024What is the Role of Small Models in the LLM Era: A Surveycited by 41
  12. 2025IRT-Router: Effective and Interpretable Multi-LLM Routing via Item Response Theorycited by 40
  13. 2025CARROT: A Cost Aware Rate Optimal Routercited by 39
  14. 2025The Avengers: A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary Giantscited by 38
  15. 2025KABB: Knowledge-Aware Bayesian Bandits for Dynamic Expert Coordination in Multi-Agent Systemscited by 37
  16. 2025Don't Overthink It: A Survey of Efficient R1-style Large Reasoning Modelscited by 37
  17. 2024Routoo: Learning to Route to Large Language Models Effectivelycited by 34
  18. 2025Beyond GPT-5: Making LLMs Cheaper and Better via Performance-Efficiency Optimized Routingcited by 34
  19. 2024Breaking the Ceiling of the LLM Community by Treating Token Generation as a Classification for Ensemblingcited by 33
  20. 2024Meta Reasoning for Large Language Modelscited by 29
  21. 2024Determine-Then-Ensemble: Necessity of Top-k Union for Large Language Model Ensemblingcited by 28
  22. 2025When One LLM Drools, Multi-LLM Collaboration Rulescited by 28
  23. 2025Causal LLM Routing: End-to-End Regret Minimization from Observational Datacited by 28
  24. 2025Capability Instruction Tuning: A New Paradigm for Dynamic LLM Routingcited by 25
  25. 2025Do We Truly Need So Many Samples? Multi-LLM Repeated Sampling Efficiently Scales Test-Time Computecited by 20
  26. 2024DAM: Dynamic Adapter Merging for Continual Video QA Learningcited by 19
  27. 2026Dynamic Model Routing and Cascading for Efficient LLM Inference: A Surveycited by 19
  28. 2025xRouter: Training Cost-Aware LLMs Orchestration System via Reinforcement Learningcited by 17
  29. 2025Cost-Aware Contrastive Routing for LLMscited by 16
  30. 2025Nature-Inspired Population-Based Evolution of Large Language Modelscited by 15
  31. 2024LEMoE: Advanced Mixture of Experts Adaptor for Lifelong Model Editing of Large Language Modelscited by 14
  32. 2024PickLLM: Context-Aware RL-Assisted Large Language Model Routingcited by 13
  33. 2025EllieSQL: Cost-Efficient Text-to-SQL with Complexity-Aware Routingcited by 13
  34. 2025Efficient Training-Free Online Routing for High-Volume Multi-LLM Servingcited by 13
  35. 2026MMR-Bench: A Comprehensive Benchmark for Multimodal LLM Routingcited by 13
  36. 2025Train-before-Test Harmonizes Language Model Rankingscited by 12
  37. 2026GraphPlanner: Graph Memory-Augmented Agentic Routing for Multi-Agent LLMscited by 12
  38. 2024LLM Chain Ensembles for Scalable and Accurate Data Annotationcited by 11

50 most cited of 172 All 172 (JSON)