# OmniRouter: Budget and Performance Controllable Multi-LLM Routing

arXiv: https://arxiv.org/abs/2502.20576 · HTML: https://mlsanity.com/paper/2502.20576/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 55 of its 73 references that are in the map.
- First posted on arXiv 27 Feb 2025.

## Builds on (most cited first, 30 of 55)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [MemGPT: Towards LLMs as Operating Systems](https://mlsanity.com/paper/2310.08560/) (2023)
- [Improving language models by retrieving from trillions of tokens](https://mlsanity.com/paper/2112.04426/) (2021)
- [Qwen2.5-Coder Technical Report](https://mlsanity.com/paper/2409.12186/) (2024)
- [A-MEM: Agentic Memory for LLM Agents](https://mlsanity.com/paper/2502.12110/) (2025)
- [Graph of Thoughts: Solving Elaborate Problems with Large Language Models](https://mlsanity.com/paper/2308.09687/) (2023)
- [RouteLLM: Learning to Route LLMs with Preference Data](https://mlsanity.com/paper/2406.18665/) (2024)
- [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://mlsanity.com/paper/2412.05579/) (2024)
- [LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset](https://mlsanity.com/paper/2309.11998/) (2023)
- [Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing](https://mlsanity.com/paper/2404.14618/) (2024)
- [DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence](https://mlsanity.com/paper/2406.11931/) (2024)
- [Llumnix: Dynamic Scheduling for Large Language Model Serving](https://mlsanity.com/paper/2406.03243/) (2024)
- [GraphRouter: A Graph-based Router for LLM Selections](https://mlsanity.com/paper/2410.03834/) (2024)
- [The Impact of Reasoning Step Length on Large Language Models](https://mlsanity.com/paper/2401.04925/) (2024)
- [S-LoRA: Serving Thousands of Concurrent LoRA Adapters](https://mlsanity.com/paper/2311.03285/) (2023)
- [CodeGen2: Lessons for Training LLMs on Programming and Natural Languages](https://mlsanity.com/paper/2305.02309/) (2023)
- [Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads](https://mlsanity.com/paper/2401.11181/) (2024)

## Built on by (most cited first, 30 of 35)

- [Two Heads are Better Than One: Test-time Scaling of Multi-agent Collaborative Reasoning](https://mlsanity.com/paper/2504.09772/) (2025)
- [R2-Router: A New Paradigm for LLM Routing with Reasoning](https://mlsanity.com/paper/2602.02823/) (2026)
- [DeepSieve: Information Sieving via LLM-as-a-Knowledge-Router](https://mlsanity.com/paper/2507.22050/) (2025)
- [Atlas: Orchestrating Heterogeneous Models and Tools for Multi-Domain Complex Reasoning](https://mlsanity.com/paper/2601.03872/) (2026)
- [SWE-Router: Routing in Multi-turn Agentic Software Engineering Tasks](https://mlsanity.com/paper/2607.00053/) (2026)
- [Budget-Aware Agentic Routing via Boundary-Guided Training](https://mlsanity.com/paper/2602.21227/) (2026)
- [Task-Aware LLM Routing with Multi-Level Task-Profile-Guided Data Synthesis for Cold-Start Scenarios](https://mlsanity.com/paper/2604.09377/) (2026)
- [Cerebrum (AIOS SDK): A Platform for Agent Development, Deployment, Distribution, and Discovery](https://mlsanity.com/paper/2503.11444/) (2025)
- [IPR: Intelligent Prompt Routing with User-Controlled Quality-Cost Trade-offs](https://mlsanity.com/paper/2509.06274/) (2025)
- [PROTEUS: SLA-Aware Routing via Lagrangian RL for Multi-LLM Serving Systems](https://mlsanity.com/paper/2601.19402/) (2026)
- [Models Under SCOPE: Scalable and Controllable Routing via Pre-hoc Reasoning](https://mlsanity.com/paper/2601.22323/) (2026)
- [ParetoBandit: Budget-Paced Adaptive Routing for Non-Stationary LLM Serving](https://mlsanity.com/paper/2604.00136/) (2026)
- [Is Escalation Worth It? On the Depth of LLM Cascades](https://mlsanity.com/paper/2605.06350/) (2026)
- [AgentSelect: Benchmark for Narrative Query-to-Agent Recommendation](https://mlsanity.com/paper/2603.03761/) (2026)
- [Robust Batch-Level Query Routing for Large Language Models under Cost and Capacity Constraints](https://mlsanity.com/paper/2603.26796/) (2026)
- [Switchcraft: AI Model Router for Agentic Tool Calling](https://mlsanity.com/paper/2605.07112/) (2026)
- [From Sampled Outcomes to Capability Distributions: Rethinking Supervision for LLM Routing](https://mlsanity.com/paper/2606.06924/) (2026)
- [WISERouter: LLM Routing with Workload Budget Constraint](https://mlsanity.com/paper/2607.23765/) (2026)
- [Node-as-Agent: Graph Agentic Network](https://mlsanity.com/paper/2508.00429/) (2025)
- [GRAIL: A Deep-Granularity Hybrid Resonance Framework for Real-Time Agent Discovery via SLM-Enhanced Indexing](https://mlsanity.com/paper/2605.02489/) (2026)
- [VLA-Pro: Cross-Task Procedural Memory Transfer for Vision-Language-Action Models](https://mlsanity.com/paper/2605.29562/) (2026)
- [STAR: Failure-Aware Markovian Routing for Multi-Agent Spatiotemporal Reasoning](https://mlsanity.com/paper/2605.10057/) (2026)
- [Latency-Quality Routing for Functionally Equivalent Tools in LLM Agents](https://mlsanity.com/paper/2605.14241/) (2026)
- [RouteJudge: An Open Platform for Reproducible and Preference-Aware LLM Routing](https://mlsanity.com/paper/2606.18774/) (2026)
- [Online Learning for Multi-Layer Hierarchical Inference under Partial and Policy-Dependent Feedback](https://mlsanity.com/paper/2603.04247/) (2026)
- [AgentOpt v0.1 Technical Report: Client-Side Optimization for LLM-Based Agent](https://mlsanity.com/paper/2604.06296/) (2026)
- [RealRoute: Dynamic Query Routing System via Retrieve-then-Verify Paradigm](https://mlsanity.com/paper/2604.20860/) (2026)
- [When Less is Enough: Efficient Inference via Collaborative Reasoning](https://mlsanity.com/paper/2605.01111/) (2026)
- [Iterative Critique-and-Routing Controller for Multi-Agent Systems with Heterogeneous LLMs](https://mlsanity.com/paper/2605.08686/) (2026)
- [Natural Language Query to Configuration for Retrieval Agents](https://mlsanity.com/paper/2605.27361/) (2026)

## Abstract (arXiv)

Large language models (LLMs) deliver superior performance but require substantial computational resources and operate with relatively low efficiency, while smaller models can efficiently handle simpler tasks with fewer resources. LLM routing is a crucial paradigm that dynamically selects the most suitable large language models from a pool of candidates to process diverse inputs, ensuring optimal resource utilization while maintaining response quality. Existing routing frameworks typically model this as a locally optimal decision-making problem, selecting the presumed best-fit LLM for each query individually, which overlooks global budget constraints, resulting in ineffective resource allocation. To tackle this problem, we introduce OmniRouter, a fundamentally controllable routing framework for multi-LLM serving. Instead of making per-query greedy choices, OmniRouter models the routing task as a constrained optimization problem, assigning models that minimize total cost while ensuring the required performance level. Specifically, a hybrid retrieval-augmented predictor is designed to predict the capabilities and costs of LLMs. After obtaining the predicted cost and performance, we utilize a constrained optimizer for cost-optimal assignments that employs Lagrangian dual decomposition with adaptive multipliers. It iteratively converges toward the globally optimal query-model allocation, dynamically balancing latency minimization against quality thresholds while adhering to heterogeneous capacity constraints. Experiments show that OmniRouter achieves up to 6.30% improvement in response accuracy while simultaneously reducing computational costs by at least 10.15% compared to competitive router baselines. The code and the dataset are available at https://github.com/dongyuanjushi/OmniRouter.