# BEST-Route: Adaptive LLM Routing with Test-Time Optimal Compute

arXiv: https://arxiv.org/abs/2506.22716 · HTML: https://mlsanity.com/paper/2506.22716/

## Key facts

- Cited by 67 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 57 references that are in the map.
- First posted on arXiv 28 Jun 2025.

## Builds on (most cited first, 30 of 41)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://mlsanity.com/paper/2408.03314/) (2024)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [Large Language Monkeys: Scaling Inference Compute with Repeated Sampling](https://mlsanity.com/paper/2407.21787/) (2024)
- [FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance](https://mlsanity.com/paper/2305.05176/) (2023)
- [RouteLLM: Learning to Route LLMs with Preference Data](https://mlsanity.com/paper/2406.18665/) (2024)
- [LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion](https://mlsanity.com/paper/2306.02561/) (2023)
- [RewardBench: Evaluating Reward Models for Language Modeling](https://mlsanity.com/paper/2403.13787/) (2024)
- [Mixture-of-Agents Enhances Large Language Model Capabilities](https://mlsanity.com/paper/2406.04692/) (2024)
- [Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts](https://mlsanity.com/paper/2406.12845/) (2024)
- [Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing](https://mlsanity.com/paper/2404.14618/) (2024)
- [Large Language Model Routing with Benchmark Datasets](https://mlsanity.com/paper/2309.15789/) (2023)
- [Speculative Decoding with Big Little Decoder](https://mlsanity.com/paper/2302.07863/) (2023)
- [Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Models](https://mlsanity.com/paper/2311.08692/) (2023)
- [Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoning](https://mlsanity.com/paper/2310.03094/) (2023)
- [Language Model Cascades: Token-level uncertainty and beyond](https://mlsanity.com/paper/2404.10136/) (2024)
- [BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Sampling](https://mlsanity.com/paper/2406.00832/) (2024)
- [Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding](https://mlsanity.com/paper/2402.12374/) (2024)

## Built on by (most cited first, 30 of 67)

- [LLMRouterBench: A Massive Benchmark and Unified Framework for LLM Routing](https://mlsanity.com/paper/2601.07206/) (2026)
- [AgentRouter: A Knowledge-Graph-Guided LLM Router for Collaborative Multi-Agent Question Answering](https://mlsanity.com/paper/2510.05445/) (2025)
- [RouterArena: An Open Platform for Comprehensive Comparison of LLM Routers](https://mlsanity.com/paper/2510.00202/) (2025)
- [Dynamic Model Routing and Cascading for Efficient LLM Inference: A Survey](https://mlsanity.com/paper/2603.04445/) (2026)
- [EvoRoute: Experience-Driven Self-Routing LLM Agent Systems](https://mlsanity.com/paper/2601.02695/) (2026)
- [NG-Router: Graph-Supervised Multi-Agent Collaboration for Nutrition Question Answering](https://mlsanity.com/paper/2510.09854/) (2025)
- [When Routing Collapses: On the Degenerate Convergence of LLM Routers](https://mlsanity.com/paper/2602.03478/) (2026)
- [Budget-Aware Agentic Routing via Boundary-Guided Training](https://mlsanity.com/paper/2602.21227/) (2026)
- [Latency and Token-Aware Test-Time Compute](https://mlsanity.com/paper/2509.09864/) (2025)
- [TRIM: Hybrid Inference via Targeted Stepwise Routing in Multi-Step Reasoning Tasks](https://mlsanity.com/paper/2601.10245/) (2026)
- [Neural Chain-of-Thought Search: Searching the Optimal Reasoning Path to Enhance Large Language Models](https://mlsanity.com/paper/2601.11340/) (2026)
- [Ares: Adaptive Reasoning Effort Selection for Efficient LLM Agents](https://mlsanity.com/paper/2603.07915/) (2026)
- [A Survey on Collaborating Small and Large Language Models for Performance, Cost-effectiveness, Cloud-edge Privacy, and Trustworthiness](https://mlsanity.com/paper/2510.13890/) (2025)
- [RACER: Risk-Aware Calibrated Efficient Routing for Large Language Models](https://mlsanity.com/paper/2603.06616/) (2026)
- [ProxRouter: Proximity-Weighted LLM Query Routing for Improved Robustness to Outliers](https://mlsanity.com/paper/2510.09852/) (2025)
- [AdaptEvolve: Improving Efficiency of Evolutionary AI Agents through Adaptive Model Selection](https://mlsanity.com/paper/2602.11931/) (2026)
- [RoboRouter: Training-Free Policy Routing for Robotic Manipulation](https://mlsanity.com/paper/2603.07892/) (2026)
- [IPR: Intelligent Prompt Routing with User-Controlled Quality-Cost Trade-offs](https://mlsanity.com/paper/2509.06274/) (2025)
- [GMTRouter: Personalized LLM Router over Multi-turn User Interactions](https://mlsanity.com/paper/2511.08590/) (2025)
- [Re-FORC: Adaptive Reward Prediction for Efficient Chain-of-Thought Reasoning](https://mlsanity.com/paper/2511.02130/) (2025)
- [Chimera: Latency- and Performance-Aware Multi-agent Serving for Heterogeneous LLMs](https://mlsanity.com/paper/2603.22206/) (2026)
- [Is Escalation Worth It? On the Depth of LLM Cascades](https://mlsanity.com/paper/2605.06350/) (2026)
- [ATLAS: Agentic Test-time Learning-to-Allocate Scaling](https://mlsanity.com/paper/2606.01667/) (2026)
- [IntroLM: Introspective Language Models via Prefilling-Time Self-Evaluation](https://mlsanity.com/paper/2601.03511/) (2026)
- [Federate the Router: Learning Language Model Routers with Sparse and Decentralized Evaluations](https://mlsanity.com/paper/2601.22318/) (2026)
- [Towards Fair and Comprehensive Evaluation of Routers in Collaborative LLM Systems](https://mlsanity.com/paper/2602.11877/) (2026)
- [Multi-Agent Reasoning Improves Compute Efficiency: Pareto-Optimal Test-Time Scaling](https://mlsanity.com/paper/2605.01566/) (2026)
- [On Time, Within Budget: Constraint-Driven Online Resource Allocation for Agentic Workflows](https://mlsanity.com/paper/2605.06110/) (2026)
- [Switchcraft: AI Model Router for Agentic Tool Calling](https://mlsanity.com/paper/2605.07112/) (2026)
- [Learning Agent Routing From Early Experience](https://mlsanity.com/paper/2605.07180/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are powerful tools but are often expensive to deploy at scale. LLM query routing mitigates this by dynamically assigning queries to models of varying cost and quality to obtain a desired trade-off. Prior query routing approaches generate only one response from the selected model and a single response from a small (inexpensive) model was often not good enough to beat a response from a large (expensive) model due to which they end up overusing the large model and missing out on potential cost savings. However, it is well known that for small models, generating multiple responses and selecting the best can enhance quality while remaining cheaper than a single large-model response. We leverage this idea to propose BEST-Route, a novel routing framework that chooses a model and the number of responses to sample from it based on query difficulty and the quality thresholds. Experiments on real-world datasets demonstrate that our method reduces costs by up to 60% with less than 1% performance drop.