# CARROT: A Cost Aware Rate Optimal Router

arXiv: https://arxiv.org/abs/2502.03261 · HTML: https://mlsanity.com/paper/2502.03261/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 49 references that are in the map.
- First posted on arXiv 5 Feb 2025.

## Builds on (most cited first, 30 of 30)

- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance](https://mlsanity.com/paper/2305.05176/) (2023)
- [LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion](https://mlsanity.com/paper/2306.02561/) (2023)
- [FinQA: A Dataset of Numerical Reasoning over Financial Data](https://mlsanity.com/paper/2109.00122/) (2021)
- [Large Language Models: A Survey](https://mlsanity.com/paper/2402.06196/) (2024)
- [TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance](https://mlsanity.com/paper/2105.07624/) (2021)
- [RouterBench: A Benchmark for Multi-LLM Routing System](https://mlsanity.com/paper/2403.12031/) (2024)
- [CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review](https://mlsanity.com/paper/2103.06268/) (2021)
- [MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoning](https://mlsanity.com/paper/2310.16049/) (2023)
- [Large Language Model Routing with Benchmark Datasets](https://mlsanity.com/paper/2309.15789/) (2023)
- [Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoning](https://mlsanity.com/paper/2310.03094/) (2023)
- [AutoMix: Automatically Mixing Language Models](https://mlsanity.com/paper/2310.12963/) (2023)
- [RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2407.11005/) (2024)
- [Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modeling](https://mlsanity.com/paper/2308.06077/) (2023)
- [Open-LLM-Leaderboard: From Multi-choice to Open-style Questions for LLMs Evaluation, Benchmark, and Arena](https://mlsanity.com/paper/2406.07545/) (2024)
- [MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures](https://mlsanity.com/paper/2406.06565/) (2024)
- [Marginal Singularity, and the Benefits of Labels in Covariate-Shift](https://mlsanity.com/paper/1803.01833/) (2018)
- [Transfer Learning for Nonparametric Classification: Minimax Rate and Adaptive Classifier](https://mlsanity.com/paper/1906.02903/) (2019)
- [Fusing Models with Complementary Expertise](https://mlsanity.com/paper/2310.01542/) (2023)
- [DelucionQA: Detecting Hallucinations in Domain-specific Question Answering](https://mlsanity.com/paper/2312.05200/) (2023)
- [MetaLLM: A High-performant and Cost-efficient Dynamic Framework for Wrapping LLMs](https://mlsanity.com/paper/2407.10834/) (2024)
- [FrugalML: How to Use ML Prediction APIs More Accurately and Cheaply](https://mlsanity.com/paper/2006.07512/) (2020)
- [The TechQA Dataset](https://mlsanity.com/paper/1911.02984/) (2019)
- [Question Answering over Electronic Devices: A New Benchmark Dataset and a Multi-Task Learning based QA Framework](https://mlsanity.com/paper/2109.05897/) (2021)
- [Minimax optimal approaches to the label shift problem in non-parametric settings](https://mlsanity.com/paper/2003.10443/) (2020)
- [Efficient Online ML API Selection for Multi-Label Classification Tasks](https://mlsanity.com/paper/2102.09127/) (2021)

## Built on by (most cited first, 30 of 39)

- [Universal Model Routing for Efficient LLM Inference](https://mlsanity.com/paper/2502.08773/) (2025)
- [OmniRouter: Budget and Performance Controllable Multi-LLM Routing](https://mlsanity.com/paper/2502.20576/) (2025)
- [Causal LLM Routing: End-to-End Regret Minimization from Observational Data](https://mlsanity.com/paper/2505.16037/) (2025)
- [Intelligence per Watt: Measuring Intelligence Efficiency of Local AI](https://mlsanity.com/paper/2511.07885/) (2025)
- [R2-Router: A New Paradigm for LLM Routing with Reasoning](https://mlsanity.com/paper/2602.02823/) (2026)
- [Efficient Training-Free Online Routing for High-Volume Multi-LLM Serving](https://mlsanity.com/paper/2509.02718/) (2025)
- [SWE-Router: Routing in Multi-turn Agentic Software Engineering Tasks](https://mlsanity.com/paper/2607.00053/) (2026)
- [Budget-Aware Agentic Routing via Boundary-Guided Training](https://mlsanity.com/paper/2602.21227/) (2026)
- [Task-Aware LLM Routing with Multi-Level Task-Profile-Guided Data Synthesis for Cold-Start Scenarios](https://mlsanity.com/paper/2604.09377/) (2026)
- [One Head, Many Models: Cross-Attention Routing for Cost-Aware LLM Selection](https://mlsanity.com/paper/2509.09782/) (2025)
- [PROTEUS: SLA-Aware Routing via Lagrangian RL for Multi-LLM Serving Systems](https://mlsanity.com/paper/2601.19402/) (2026)
- [Scaling Small Agents Through Strategy Auctions](https://mlsanity.com/paper/2602.02751/) (2026)
- [Switchcraft: AI Model Router for Agentic Tool Calling](https://mlsanity.com/paper/2605.07112/) (2026)
- [Cost-Optimal LLM Routing with Limited User Feedback under User Satisfaction Guarantees](https://mlsanity.com/paper/2606.19376/) (2026)
- [Correlation-Aware Contextual Bandits with Surrogate Rewards for LLM Routing](https://mlsanity.com/paper/2607.09015/) (2026)
- [Engineering Reliable Coding Agents: Evaluating and Operating the System Around the Model](https://mlsanity.com/paper/2608.13867/) (2026)
- [Rich Insights from Cheap Signals: Efficient Evaluations via Tensor Factorization](https://mlsanity.com/paper/2603.02029/) (2026)
- [Markets, Not Planners: Decentralized Orchestration of LLM Agents with Private Information](https://mlsanity.com/paper/2608.23867/) (2026)
- [Meta-Router: Bridging Gold-standard and Preference-based Evaluations in Large Language Model Routing](https://mlsanity.com/paper/2509.25535/) (2025)
- [The Routing Plateau: Understanding the Accuracy Limits of LLM Routers](https://mlsanity.com/paper/2606.07587/) (2026)
- [Training-Free versus Training-Based Intent Classification in LLMs: Accuracy, Robustness, and Failure Modes](https://mlsanity.com/paper/2608.02415/) (2026)
- [Pre-Inference Routing for Cost-Efficient Document Field Extraction](https://mlsanity.com/paper/2608.06607/) (2026)
- [BudgetVerify: Budget-Tiered Verification for Financial QA](https://mlsanity.com/paper/2609.33052/) (2026)
- [Dynamic LLM Routers are Often Misguided](https://mlsanity.com/paper/2610.02762/) (2026)
- [Decision-Centric Design for LLM Systems](https://mlsanity.com/paper/2604.00414/) (2026)
- [Flexible Routing via Uncertainty Decomposition](https://mlsanity.com/paper/2605.07805/) (2026)
- [Capturing LLM Capabilities via Evidence-Calibrated Query Clustering](https://mlsanity.com/paper/2605.17110/) (2026)
- [Natural Language Query to Configuration for Retrieval Agents](https://mlsanity.com/paper/2605.27361/) (2026)
- [SCOPE-Router: Cost-Aware Open-Set VLM Routing for Execution-Oriented Tasks](https://mlsanity.com/paper/2608.12127/) (2026)
- [LLMs Can Predict Failure Risk, But Struggle to Predict Which Collaboration Protocol Pays Off: Cost-Aware Protocol Routing Across Reasoning Tasks](https://mlsanity.com/paper/2608.14927/) (2026)

## Abstract (arXiv)

With the rapid growth in the number of Large Language Models (LLMs), there has been a recent interest in LLM routing, or directing queries to the cheapest LLM that can deliver a suitable response. We conduct a minimax analysis of the routing problem, providing a lower bound and finding that a simple router that predicts both cost and accuracy for each question can be minimax optimal. Inspired by this, we introduce CARROT, a Cost AwaRe Rate Optimal rouTer that selects a model based on estimates of the models' cost and performance. Alongside CARROT, we also introduce the Smart Price-aware ROUTing (SPROUT) dataset to facilitate routing on a wide spectrum of queries with the latest state-of-the-art LLMs. Using SPROUT and prior benchmarks such as Routerbench and open-LLM-leaderboard-v2 we empirically validate CARROT's performance against several alternative routers.