mlsanityData to 7 Oct 2026

Universal Model Routing for Efficient LLM Inference

arXiv 2502.08773 ↗First posted 12 Feb 2025cs.CL cs.LGcited by 80

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Model routing is a simple technique for reducing the inference cost of large language models (LLMs), wherein one maintains a pool of candidate LLMs, and learns to route each prompt to the smallest feasible LLM. Existing works focus on learning a router for a fixed pool of LLMs. In this paper, we consider the problem of dynamic routing, where new, previously unobserved LLMs are available at test time. We propose UniRoute, a new approach to this problem that relies on representing each LLM as a feature vector, derived based on predictions on a set of representative prompts. Based on this, we detail two effective instantiations of UniRoute, relying on cluster-based routing and a learned cluster map respectively. We show that these are estimates of a theoretically optimal routing rule, and quantify their errors via an excess risk bound. Experiments on a range of public benchmarks show the effectiveness of UniRoute in routing amongst more than 30 unseen LLMs.

Key facts. Cited by 80 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 92 of its 129 references that are in the map. First posted on arXiv 12 Feb 2025.

Builds on

92 of 129 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024TensorOpera Router: A Multi-Model Router for Efficient LLM Inference22 Aug
  2. 2024A Survey on Efficient Inference for Large Language Models22 Apr

Other references · 90

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  8. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  9. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  10. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  11. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  12. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
38 more
  1. 2018Hierarchical Neural Story Generationcited by 1,182
  2. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  3. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  4. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  5. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  6. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  7. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  8. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 633
  9. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  10. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
  11. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
  12. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  13. 2022Efficiently Scaling Transformer Inferencecited by 529
  14. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  15. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 462
  16. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
  17. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 362
  18. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 354
  19. 2023How Good Are GPT Models at Machine Translation? A Comprehensive Evaluationcited by 344
  20. 2019SelectiveNet: A Deep Neural Network with an Integrated Reject Optioncited by 333
  21. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
  22. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 302
  23. 2022Confident Adaptive Language Modelingcited by 297
  24. 2023Improving Text Embeddings with Large Language Modelscited by 295
  25. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
  26. 2020Consistent Estimators for Learning to Defer to an Expertcited by 262
  27. 2017Predict Responsibly: Improving Fairness and Accuracy by Learning to Defercited by 228
  28. 2024RouterBench: A Benchmark for Multi-LLM Routing Systemcited by 222
  29. 2019Model Fusion via Optimal Transportcited by 217
  30. 2019Task2Vec: Task Embedding for Meta-Learningcited by 197
  31. 2023Efficient Large Language Models: A Surveycited by 183
  32. 2023Large Language Model Routing with Benchmark Datasetscited by 166
  33. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillationcited by 160
  34. 2024GraphRouter: A Graph-based Router for LLM Selectionscited by 154
  35. 2021Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Bettercited by 153
  36. 2017Controlling Linguistic Style Aspects in Neural Language Generationcited by 147
  37. 2023Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Modelscited by 141
  38. 2023Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoningcited by 140

50 most cited of 90 All 90 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 25; 2026: 55.

80 papers.

Sort:

By area:

  1. 2025RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMscited by 58
  2. 2025The Avengers: A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary Giantscited by 37
  3. 2026LLMRouterBench: A Massive Benchmark and Unified Framework for LLM Routingcited by 35
  4. 2025ICL-Router: In-Context Learned Model Representations for LLM Routingcited by 26
  5. 2025Route to Reason: Adaptive Routing for LLM and Reasoning Strategy Selectioncited by 23
  6. 2025RouterArena: An Open Platform for Comprehensive Comparison of LLM Routerscited by 19
  7. 2025Online Multi-LLM Selection via Contextual Bandits under Unstructured Context Evolutioncited by 18
  8. 2025Cost-Aware Contrastive Routing for LLMscited by 16
  9. 2026Dynamic Model Routing and Cascading for Efficient LLM Inference: A Surveycited by 16
  10. 2025Efficient Training-Free Online Routing for High-Volume Multi-LLM Servingcited by 13
  11. 2026EvoRoute: Experience-Driven Self-Routing LLM Agent Systemscited by 13
  12. 2026R2-Router: A New Paradigm for LLM Routing with Reasoningcited by 13
38 more
  1. 2025Mixture of Thoughts: Learning to Aggregate What Experts Think, Not Just What They Saycited by 12
  2. 2025Arch-Router: Aligning LLM Routing with Human Preferencescited by 9
  3. 2025Learning Compact Representations of LLM Abilities via Item Response Theorycited by 9
  4. 2026SWE-Router: Routing in Multi-turn Agentic Software Engineering Taskscited by 9
  5. 2025Semantic Agreement Enables Efficient Open-Ended LLM Cascadescited by 7
  6. 2026V0: A Generalist Value Model for Any Policy at State Zerocited by 7
  7. 2026Ares: Adaptive Reasoning Effort Selection for Efficient LLM Agentscited by 7
  8. 2026The Replay Gap: Static Evaluation of Model Switching in LLM Agents Scores the Wrong Worldcited by 7
  9. 2025LLM Routing with Dueling Feedbackcited by 6
  10. 2026RoboRouter: Training-Free Policy Routing for Robotic Manipulationcited by 6
  11. 2025IPR: Intelligent Prompt Routing with User-Controlled Quality-Cost Trade-offscited by 5
  12. 2025ProxRouter: Proximity-Weighted LLM Query Routing for Improved Robustness to Outlierscited by 5
  13. 2025Adversarial Robustness in One-Stage Learning-to-Defercited by 5
  14. 2025Cost-Aware Routing for Efficient Text-To-Image Generationcited by 4
  15. 2025One Head, Many Models: Cross-Attention Routing for Cost-Aware LLM Selectioncited by 4
  16. 2025Re-FORC: Adaptive Reward Prediction for Efficient Chain-of-Thought Reasoningcited by 4
  17. 2026Routing with Generated Data: Annotation-Free LLM Skill Estimation and Expert Selectioncited by 4
  18. 2026LAPS: A Length-Aware-Prefill LLM Serving Systemcited by 4
  19. 2026Chimera: Latency- and Performance-Aware Multi-agent Serving for Heterogeneous LLMscited by 4
  20. 2026GenRouter: Unified Workflow Routing for Agentic Image Generationcited by 4
  21. 2025Near-Optimal Online Deployment and Routing for Streaming LLMscited by 3
  22. 2026TCAndon-Router: Adaptive Reasoning Router for Multi-Agent Collaborationcited by 3
  23. 2026SWE-Prot'eg'e: Learning to Selectively Collaborate With an Expert Unlocks Small Language Models as Software Engineering Agentscited by 3
  24. 2026AgentSelect: Benchmark for Narrative Query-to-Agent Recommendationcited by 3
  25. 2026Switchcraft: AI Model Router for Agentic Tool Callingcited by 3
  26. 2026Learning Agent Routing From Early Experiencecited by 3
  27. 2025Resource-efficient Inference with Foundation Model Programscited by 2
  28. 2026Minimizing Mismatch Risk: A Prototype-Based Routing Framework for Zero-shot LLM-generated Text Detectioncited by 2
  29. 2026Interfaze: The Future of AI is built on Task-Specific Small Modelscited by 2
  30. 2026When More Experts Hurt: Underfitting in Multi-Expert Learning to Defercited by 2
  31. 2026Robust Batch-Level Query Routing for Large Language Models under Cost and Capacity Constraintscited by 2
  32. 2025CONCUR: A Framework for Continual Constrained and Unconstrained Routingcited by 1
  33. 2026Privacy-Preserving LLMs Routingcited by 1
  34. 2026Latency-Quality Routing for Functionally Equivalent Tools in LLM Agentscited by 1
  35. 2026FlyRoute: Self-Evolving Agent Profiling via Data Flywheel for Adaptive Task Routingcited by 1
  36. 2026Policy Regret for Embedding Model Routing: Contextual Bandits with Low-Rank Expertscited by 1
  37. 2026RouteBalance: Fused Model Routing and Load Balancing for Heterogeneous LLM Servingcited by 1
  38. 2026Learning to Prompt: Improving Student Engagement with Adaptive LLM-based High-School Tutoringcited by 1

50 most cited of 80 All 80 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Rethinking Predictive Modeling for LLM Routing: When Simple kNN Beats Complex Learned Routers
  2. 2025Lookahead Routing for Large Language Models
  3. 2025Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniques
  4. 2025Confidence-Guided Stepwise Model Routing for Cost-Efficient Reasoning
  5. 2025Adaptive LLM Routing under Budget Constraints
  6. 2025Dynamic LLM Routing and Selection based on User Preferences: Balancing Performance, Cost, and Ethics
  7. 2024Routing Experts: Learning to Route Dynamic Experts in Multi-modal Large Language Models
  8. 2026Dynamic Routing as a New Dimension for Test-time Versatility of LLMs