# Adaptive LLM Routing under Budget Constraints

arXiv: https://arxiv.org/abs/2508.21141 · HTML: https://mlsanity.com/paper/2508.21141/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 33 references that are in the map.
- First posted on arXiv 28 Aug 2025.

## Builds on (most cited first, 20 of 20)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance](https://mlsanity.com/paper/2305.05176/) (2023)
- [RouteLLM: Learning to Route LLMs with Preference Data](https://mlsanity.com/paper/2406.18665/) (2024)
- [LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion](https://mlsanity.com/paper/2306.02561/) (2023)
- [Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing](https://mlsanity.com/paper/2404.14618/) (2024)
- [What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning](https://mlsanity.com/paper/2312.15685/) (2023)
- [One Embedder, Any Task: Instruction-Finetuned Text Embeddings](https://mlsanity.com/paper/2212.09741/) (2022)
- [Simple linear attention language models balance the recall-throughput tradeoff](https://mlsanity.com/paper/2402.18668/) (2024)
- [A Careful Examination of Large Language Model Performance on Grade School Arithmetic](https://mlsanity.com/paper/2405.00332/) (2024)
- [GraphRouter: A Graph-based Router for LLM Selections](https://mlsanity.com/paper/2410.03834/) (2024)
- [Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modeling](https://mlsanity.com/paper/2308.06077/) (2023)
- [Learning to Route LLMs with Confidence Tokens](https://mlsanity.com/paper/2410.13284/) (2024)
- [TensorOpera Router: A Multi-Model Router for Efficient LLM Inference](https://mlsanity.com/paper/2408.12320/) (2024)
- [A Survey on Practical Applications of Multi-Armed and Contextual Bandits](https://mlsanity.com/paper/1904.10040/) (2019)
- [MetaLLM: A High-performant and Cost-efficient Dynamic Framework for Wrapping LLMs](https://mlsanity.com/paper/2407.10834/) (2024)
- [Blending Is All You Need: Cheaper, Better Alternative to Trillion-Parameters LLM](https://mlsanity.com/paper/2401.02994/) (2024)
- [Efficient Online ML API Selection for Multi-Label Classification Tasks](https://mlsanity.com/paper/2102.09127/) (2021)

## Built on by (most cited first, 24 of 24)

- [Dynamic Model Routing and Cascading for Efficient LLM Inference: A Survey](https://mlsanity.com/paper/2603.04445/) (2026)
- [Budget-Aware Agentic Routing via Boundary-Guided Training](https://mlsanity.com/paper/2602.21227/) (2026)
- [Adaptive Test-Time Compute Allocation for Reasoning LLMs via Constrained Policy Optimization](https://mlsanity.com/paper/2604.14853/) (2026)
- [BOute: Cost-Efficient LLM Serving with Heterogeneous LLMs and GPUs via Multi-Objective Bayesian Optimization](https://mlsanity.com/paper/2602.10729/) (2026)
- [One Head, Many Models: Cross-Attention Routing for Cost-Aware LLM Selection](https://mlsanity.com/paper/2509.09782/) (2025)
- [Reference-Free Rating of LLM Responses via Latent Information](https://mlsanity.com/paper/2509.24678/) (2025)
- [ParetoBandit: Budget-Paced Adaptive Routing for Non-Stationary LLM Serving](https://mlsanity.com/paper/2604.00136/) (2026)
- [Agentic Routing: The Harness-Native Data Flywheel](https://mlsanity.com/paper/2607.11399/) (2026)
- [Learning Agent Routing From Early Experience](https://mlsanity.com/paper/2605.07180/) (2026)
- [WISERouter: LLM Routing with Workload Budget Constraint](https://mlsanity.com/paper/2607.23765/) (2026)
- [AgentRouter: Heterogeneous Model Routing for Cost-Optimal Multi-Step Agentic Workflows](https://mlsanity.com/paper/2609.22951/) (2026)
- [A Component-Based Survey of Interactions between Large Language Models and Multi-Armed Bandits](https://mlsanity.com/paper/2601.12945/) (2026)
- [Zero-Shot Confidence Estimation for Small LLMs: When Supervised Baselines Aren't Worth Training](https://mlsanity.com/paper/2605.02241/) (2026)
- [Retrieval-Conditioned Topology Selection with Provable Budget Conservation for Multi-Agent Code Generation](https://mlsanity.com/paper/2605.05657/) (2026)
- [RouteBalance: Fused Model Routing and Load Balancing for Heterogeneous LLM Serving](https://mlsanity.com/paper/2606.17949/) (2026)
- [Optimal and Efficient Contextual Combinatorial Semi-bandits with General Function Approximation](https://mlsanity.com/paper/2607.13686/) (2026)
- [Steering Frozen LLMs: Adaptive Social Alignment via Online Prompt Routing](https://mlsanity.com/paper/2603.15647/) (2026)
- [ReLope: KL-Regularized LoRA Probes for Multimodal LLM Routing](https://mlsanity.com/paper/2603.24787/) (2026)
- [Learning to Assign Prediction Tasks to Agents with Capacity Constraints](https://mlsanity.com/paper/2605.27999/) (2026)
- [RLCascadeRouter: Quality-Estimator-Free Cascade Routing via Reinforcement Learning](https://mlsanity.com/paper/2608.15817/) (2026)
- [Bandits with Multiple Optimal Arms: Minimax Regret and Non-Adaptivity](https://mlsanity.com/paper/2609.38659/) (2026)
- [JOVE: Joint Execution and Verification for Resource-Aware LLM Task Graphs](https://mlsanity.com/paper/2610.03296/) (2026)
- [Fusion is the New Mutation: Bandit-Guided Evolution on Workflow Graphs](https://mlsanity.com/paper/2610.05284/) (2026)
- [TurnCost: Turns, Not Observations: Cache-Aware Loop Engineering for Coding Agents](https://mlsanity.com/paper/2610.15093/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have revolutionized natural language processing, but their varying capabilities and costs pose challenges in practical applications. LLM routing addresses this by dynamically selecting the most suitable LLM for each query/task. Previous approaches treat this as a supervised learning problem, assuming complete knowledge of optimal query-LLM pairings. However, real-world scenarios lack such comprehensive mappings and face evolving user queries. We thus propose to study LLM routing as a contextual bandit problem, enabling adaptive decision-making using bandit feedback without requiring exhaustive inference across all LLMs for all queries (in contrast to supervised routing). To address this problem, we develop a shared embedding space for queries and LLMs, where query and LLM embeddings are aligned to reflect their affinity. This space is initially learned from offline human preference data and refined through online bandit feedback. We instantiate this idea through Preference-prior Informed Linucb fOr adaptive rouTing (PILOT), a novel extension of LinUCB. To handle diverse user budgets for model routing, we introduce an online cost policy modeled as a multi-choice knapsack problem, ensuring resource-efficient routing.