mlsanityData to 14 Oct 2026

BEST-Route: Adaptive LLM Routing with Test-Time Optimal Compute

arXiv 2506.22716 ↗First posted 28 Jun 2025cs.LG cs.AI cs.CLcited by 67

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) are powerful tools but are often expensive to deploy at scale. LLM query routing mitigates this by dynamically assigning queries to models of varying cost and quality to obtain a desired trade-off. Prior query routing approaches generate only one response from the selected model and a single response from a small (inexpensive) model was often not good enough to beat a response from a large (expensive) model due to which they end up overusing the large model and missing out on potential cost savings. However, it is well known that for small models, generating multiple responses and selecting the best can enhance quality while remaining cheaper than a single large-model response. We leverage this idea to propose BEST-Route, a novel routing framework that chooses a model and the number of responses to sample from it based on query difficulty and the quality thresholds. Experiments on real-world datasets demonstrate that our method reduces costs by up to 60% with less than 1% performance drop.

Key facts. Cited by 67 papers in the map (data through 14 Oct 2026). Builds on 41 of its 57 references that are in the map. First posted on arXiv 28 Jun 2025.

Builds on

41 of 57 references.

References · 41

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  7. 2020Learning to summarize from human feedbackcited by 2,588
  8. 2023A Survey of Large Language Modelscited by 2,576
  9. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  10. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  11. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  12. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
29 more
  1. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  2. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  3. 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
  4. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  5. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
  6. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
  7. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  8. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  9. 2024Mixture-of-Agents Enhances Large Language Model Capabilitiescited by 438
  10. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 376
  11. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 372
  12. 2023Large Language Model Routing with Benchmark Datasetscited by 172
  13. 2023Speculative Decoding with Big Little Decodercited by 164
  14. 2023Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Modelscited by 145
  15. 2023Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoningcited by 143
  16. 2024Language Model Cascades: Token-level uncertainty and beyondcited by 121
  17. 2024BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Samplingcited by 121
  18. 2024Sequoia: Scalable, Robust, and Hardware-aware Speculative Decodingcited by 103
  19. 2023Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modelingcited by 95
  20. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  21. 2024Pack of LLMs: Model Fusion at Test-Time via Perplexity Optimizationcited by 56
  22. 2024Offline Training of Language Model Agents with Functions as Learnable Weightscited by 48
  23. 2024Harnessing the Power of Multiple Minds: Lessons Learned from LLM Routingcited by 46
  24. 2024Faster Cascades via Speculative Decodingcited by 38
  25. 2022A global analysis of metrics used for measuring performance in natural language processingcited by 17
  26. 2023Expand, Rerank, and Retrieve: Query Reranking for Open-Domain Question Answeringcited by 15
  27. 2024EcoAct: Economic Agent Determines When to Register What Actioncited by 11
  28. 2022On Efficient Approximate Queries over Machine Learning Modelscited by 7
  29. 2024OCCAM: Towards Cost-Efficient and Accuracy-Aware Classification Inferencecited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 57.

67 papers.

Sort:

By area:

  1. 2026LLMRouterBench: A Massive Benchmark and Unified Framework for LLM Routingcited by 38
  2. 2025AgentRouter: A Knowledge-Graph-Guided LLM Router for Collaborative Multi-Agent Question Answeringcited by 32
  3. 2025RouterArena: An Open Platform for Comprehensive Comparison of LLM Routerscited by 22
  4. 2026Dynamic Model Routing and Cascading for Efficient LLM Inference: A Surveycited by 19
  5. 2026EvoRoute: Experience-Driven Self-Routing LLM Agent Systemscited by 16
  6. 2025NG-Router: Graph-Supervised Multi-Agent Collaboration for Nutrition Question Answeringcited by 10
  7. 2026When Routing Collapses: On the Degenerate Convergence of LLM Routerscited by 10
  8. 2026Budget-Aware Agentic Routing via Boundary-Guided Trainingcited by 10
  9. 2025Latency and Token-Aware Test-Time Computecited by 8
  10. 2026TRIM: Hybrid Inference via Targeted Stepwise Routing in Multi-Step Reasoning Taskscited by 8
  11. 2026Neural Chain-of-Thought Search: Searching the Optimal Reasoning Path to Enhance Large Language Modelscited by 8
  12. 2026Ares: Adaptive Reasoning Effort Selection for Efficient LLM Agentscited by 8
38 more
  1. 2025A Survey on Collaborating Small and Large Language Models for Performance, Cost-effectiveness, Cloud-edge Privacy, and Trustworthinesscited by 7
  2. 2026RACER: Risk-Aware Calibrated Efficient Routing for Large Language Modelscited by 7
  3. 2025ProxRouter: Proximity-Weighted LLM Query Routing for Improved Robustness to Outlierscited by 6
  4. 2026AdaptEvolve: Improving Efficiency of Evolutionary AI Agents through Adaptive Model Selectioncited by 6
  5. 2026RoboRouter: Training-Free Policy Routing for Robotic Manipulationcited by 6
  6. 2025IPR: Intelligent Prompt Routing with User-Controlled Quality-Cost Trade-offscited by 5
  7. 2025GMTRouter: Personalized LLM Router over Multi-turn User Interactionscited by 5
  8. 2025Re-FORC: Adaptive Reward Prediction for Efficient Chain-of-Thought Reasoningcited by 4
  9. 2026Chimera: Latency- and Performance-Aware Multi-agent Serving for Heterogeneous LLMscited by 4
  10. 2026Is Escalation Worth It? On the Depth of LLM Cascadescited by 4
  11. 2026ATLAS: Agentic Test-time Learning-to-Allocate Scalingcited by 4
  12. 2026IntroLM: Introspective Language Models via Prefilling-Time Self-Evaluationcited by 3
  13. 2026Federate the Router: Learning Language Model Routers with Sparse and Decentralized Evaluationscited by 3
  14. 2026Towards Fair and Comprehensive Evaluation of Routers in Collaborative LLM Systemscited by 3
  15. 2026Multi-Agent Reasoning Improves Compute Efficiency: Pareto-Optimal Test-Time Scalingcited by 3
  16. 2026On Time, Within Budget: Constraint-Driven Online Resource Allocation for Agentic Workflowscited by 3
  17. 2026Switchcraft: AI Model Router for Agentic Tool Callingcited by 3
  18. 2026Learning Agent Routing From Early Experiencecited by 3
  19. 2026From Sampled Outcomes to Capability Distributions: Rethinking Supervision for LLM Routingcited by 3
  20. 2026WISERouter: LLM Routing with Workload Budget Constraintcited by 3
  21. 2025Astrolabe: Balancing Load in LLM Serving with Randomized Prediction-Guided Schedulingcited by 2
  22. 2026Dynamic Mixed-Precision Routing for Efficient Multi-step LLM Interactioncited by 2
  23. 2026When Do LLMs Reason? A Dynamical Systems View via Entropy Phase Transitionscited by 2
  24. 2026SeqRoute: Global Budget-Aware Sequential LLM Routing via Offline Reinforcement Learningcited by 2
  25. 2026Uncertainty Is Not Enough: Value-of-Information Routing for Mixtures of LoRA Expertscited by 2
  26. 2026Cost-Utility Alignment in LLM Agent Trajectories:Profiling,Attribution,Diagnosis,Adaptation,and Evaluationcited by 2
  27. 2026CR^2: Cost-Aware Risk-Controlled Routing for Wireless Device-Edge LLM Inferencecited by 1
  28. 2026Latency-Quality Routing for Functionally Equivalent Tools in LLM Agentscited by 1
  29. 2026UniScale: Adaptive Unified Inference Scaling via Online Joint Optimization of Model Routing and Test-Time Scalingcited by 1
  30. 2026RouteBalance: Fused Model Routing and Load Balancing for Heterogeneous LLM Servingcited by 1
  31. 2026The Capability Frontier: Benchmarks Miss 82% of Model Performancecited by 1
  32. 2026Resample or Reroute? Recoverable Stopping Debt Without Identified Action Selectioncited by 1
  33. 2026Mind the Delay: Latency-aware Query Routing for Heterogeneous LLM poolscited by 1
  34. 2026Pre-Inference Routing for Cost-Efficient Document Field Extractioncited by 1
  35. 2026MERA: Model Evolution and Routing with Skill Adaptation for Agentic Systems at Scalecited by 1
  36. 2026ReLope: KL-Regularized LoRA Probes for Multimodal LLM Routingcited by 0
  37. 2026RouterWise: Joint Resource Allocation and Routing for Latency-Aware Multi-Model LLM Servingcited by 0
  38. 2026Flexible Routing via Uncertainty Decompositioncited by 0

50 most cited of 67 All 67 (JSON)