mlsanityData to 7 Oct 2026

CARROT: A Cost Aware Rate Optimal Router

arXiv 2502.03261 ↗First posted 5 Feb 2025stat.ML cs.LG cs.NIcited by 34

Not on the map. Why?

Abstract (arXiv)

With the rapid growth in the number of Large Language Models (LLMs), there has been a recent interest in LLM routing, or directing queries to the cheapest LLM that can deliver a suitable response. We conduct a minimax analysis of the routing problem, providing a lower bound and finding that a simple router that predicts both cost and accuracy for each question can be minimax optimal. Inspired by this, we introduce CARROT, a Cost AwaRe Rate Optimal rouTer that selects a model based on estimates of the models' cost and performance. Alongside CARROT, we also introduce the Smart Price-aware ROUTing (SPROUT) dataset to facilitate routing on a wide spectrum of queries with the latest state-of-the-art LLMs. Using SPROUT and prior benchmarks such as Routerbench and open-LLM-leaderboard-v2 we empirically validate CARROT's performance against several alternative routers.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Builds on 30 of its 49 references that are in the map. First posted on arXiv 5 Feb 2025.

Builds on

30 of 49 references.

References · 30

  1. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  2. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  3. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  4. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  5. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  6. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  7. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 552
  8. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  9. 2024Large Language Models: A Surveycited by 506
  10. 2021TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Financecited by 402
  11. 2021CUAD: An Expert-Annotated NLP Dataset for Legal Contract Reviewcited by 232
  12. 2024RouterBench: A Benchmark for Multi-LLM Routing Systemcited by 222
18 more
  1. 2023MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoningcited by 195
  2. 2023Large Language Model Routing with Benchmark Datasetscited by 166
  3. 2023Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoningcited by 140
  4. 2023AutoMix: Automatically Mixing Language Modelscited by 116
  5. 2024RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systemscited by 115
  6. 2023Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modelingcited by 94
  7. 2024Open-LLM-Leaderboard: From Multi-choice to Open-style Questions for LLMs Evaluation, Benchmark, and Arenacited by 76
  8. 2024MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixturescited by 73
  9. 2018Marginal Singularity, and the Benefits of Labels in Covariate-Shiftcited by 65
  10. 2019Transfer Learning for Nonparametric Classification: Minimax Rate and Adaptive Classifiercited by 64
  11. 2023Fusing Models with Complementary Expertisecited by 45
  12. 2023DelucionQA: Detecting Hallucinations in Domain-specific Question Answeringcited by 43
  13. 2024MetaLLM: A High-performant and Cost-efficient Dynamic Framework for Wrapping LLMscited by 39
  14. 2020FrugalML: How to Use ML Prediction APIs More Accurately and Cheaplycited by 33
  15. 2019The TechQA Datasetcited by 29
  16. 2021Question Answering over Electronic Devices: A New Benchmark Dataset and a Multi-Task Learning based QA Frameworkcited by 21
  17. 2020Minimax optimal approaches to the label shift problem in non-parametric settingscited by 20
  18. 2021Efficient Online ML API Selection for Multi-Label Classification Taskscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 27.

34 papers.

Sort:

By area:

  1. 2025Universal Model Routing for Efficient LLM Inferencecited by 80
  2. 2025OmniRouter: Budget and Performance Controllable Multi-LLM Routingcited by 35
  3. 2025Causal LLM Routing: End-to-End Regret Minimization from Observational Datacited by 27
  4. 2025Efficient Training-Free Online Routing for High-Volume Multi-LLM Servingcited by 13
  5. 2025Intelligence per Watt: Measuring Intelligence Efficiency of Local AIcited by 13
  6. 2026R2-Router: A New Paradigm for LLM Routing with Reasoningcited by 13
  7. 2026SWE-Router: Routing in Multi-turn Agentic Software Engineering Taskscited by 9
  8. 2026Budget-Aware Agentic Routing via Boundary-Guided Trainingcited by 8
  9. 2026Task-Aware LLM Routing with Multi-Level Task-Profile-Guided Data Synthesis for Cold-Start Scenarioscited by 8
  10. 2025One Head, Many Models: Cross-Attention Routing for Cost-Aware LLM Selectioncited by 4
  11. 2026PROTEUS: SLA-Aware Routing via Lagrangian RL for Multi-LLM Serving Systemscited by 4
  12. 2026Scaling Small Agents Through Strategy Auctionscited by 3
22 more
  1. 2026Switchcraft: AI Model Router for Agentic Tool Callingcited by 3
  2. 2025Meta-Router: Bridging Gold-standard and Preference-based Evaluations in Large Language Model Routingcited by 1
  3. 2026Rich Insights from Cheap Signals: Efficient Evaluations via Tensor Factorizationcited by 1
  4. 2026Cost-Optimal LLM Routing with Limited User Feedback under User Satisfaction Guaranteescited by 1
  5. 2026Pre-Inference Routing for Cost-Efficient Document Field Extractioncited by 1
  6. 2026Engineering Reliable Coding Agents: Evaluating and Operating the System Around the Modelcited by 1
  7. 2026Markets, Not Planners: Decentralized Orchestration of LLM Agents with Private Informationcited by 1
  8. 2026Decision-Centric Design for LLM Systemscited by 0
  9. 2026Flexible Routing via Uncertainty Decompositioncited by 0
  10. 2026Capturing LLM Capabilities via Evidence-Calibrated Query Clusteringcited by 0
  11. 2026Natural Language Query to Configuration for Retrieval Agentscited by 0
  12. 2026The Routing Plateau: Understanding the Accuracy Limits of LLM Routerscited by 0
  13. 2026Correlation-Aware Contextual Bandits with Surrogate Rewards for LLM Routingcited by 0
  14. 2026Training-Free versus Training-Based Intent Classification in LLMs: Accuracy, Robustness, and Failure Modescited by 0
  15. 2026SCOPE-Router: Cost-Aware Open-Set VLM Routing for Execution-Oriented Taskscited by 0
  16. 2026LLMs Can Predict Failure Risk, But Struggle to Predict Which Collaboration Protocol Pays Off: Cost-Aware Protocol Routing Across Reasoning Taskscited by 0
  17. 2026Learning to Grade Efficiently: A Bandit-Driven Prompt-Selection Framework for Low-Cost LLM Essay Scoringcited by 0
  18. 2026Atlas: Optimizing Deployment of Compound AI Workflows on Heterogeneous Clusterscited by 0
  19. 2026When Does a Skill Add Value? Task-Conditional Gain Prediction for Selective Skill Usecited by 0
  20. 2026BudgetVerify: Budget-Tiered Verification for Financial QAcited by 0
  21. 2026Routing Without Embeddings: Fast And Interpretable Routing With Regular Expressionscited by 0
  22. 2026You Cannot Pick a Provider From the Price List: Market-Aware Routing for Open-Weight LLM Inferencecited by 0