# Learning to Route LLMs from Bandit Feedback: One Policy, Many Trade-offs

arXiv: https://arxiv.org/abs/2510.07429 · HTML: https://mlsanity.com/paper/2510.07429/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 36 references that are in the map.
- First posted on arXiv 8 Oct 2025.

## Builds on (most cited first, 23 of 23)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://mlsanity.com/paper/2212.03533/) (2022)
- [Yi: Open Foundation Models by 01.AI](https://mlsanity.com/paper/2403.04652/) (2024)
- [RouterBench: A Benchmark for Multi-LLM Routing System](https://mlsanity.com/paper/2403.12031/) (2024)
- [Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Sampling](https://mlsanity.com/paper/1802.09127/) (2018)
- [GraphRouter: A Graph-based Router for LLM Selections](https://mlsanity.com/paper/2410.03834/) (2024)
- [RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Models](https://mlsanity.com/paper/2409.19886/) (2024)
- [Cost-Effective Online Multi-LLM Selection with Versatile Reward Models](https://mlsanity.com/paper/2405.16587/) (2024)
- [LLM Bandit: Cost-Efficient LLM Generation via Preference-Conditioned Dynamic Routing](https://mlsanity.com/paper/2502.02743/) (2025)
- [Doing More with Less: A Survey on Routing Strategies for Resource Optimisation in Large Language Model-Based Systems](https://mlsanity.com/paper/2502.00409/) (2025)

## Built on by (most cited first, 15 of 15)

- [PROTEUS: SLA-Aware Routing via Lagrangian RL for Multi-LLM Serving Systems](https://mlsanity.com/paper/2601.19402/) (2026)
- [ParetoBandit: Budget-Paced Adaptive Routing for Non-Stationary LLM Serving](https://mlsanity.com/paper/2604.00136/) (2026)
- [Correlation-Aware Contextual Bandits with Surrogate Rewards for LLM Routing](https://mlsanity.com/paper/2607.09015/) (2026)
- [Beyond Top-k Skill Retrieval: Diversity-Aware Skill Routing for LLM Agents](https://mlsanity.com/paper/2609.05824/) (2026)
- [A Component-Based Survey of Interactions between Large Language Models and Multi-Armed Bandits](https://mlsanity.com/paper/2601.12945/) (2026)
- [Cascade-Aware Multi-Agent Routing: Spatio-Temporal Sidecars and Geometry-Switching](https://mlsanity.com/paper/2603.17112/) (2026)
- [Zero-Shot Confidence Estimation for Small LLMs: When Supervised Baselines Aren't Worth Training](https://mlsanity.com/paper/2605.02241/) (2026)
- [Latency-Quality Routing for Functionally Equivalent Tools in LLM Agents](https://mlsanity.com/paper/2605.14241/) (2026)
- [FlexRouter: Learning Complementary Model Sets for Flexible LLM Routing](https://mlsanity.com/paper/2609.38585/) (2026)
- [POLAR: Online Learning for LoRA Adapter Caching and Routing in Edge LLM Serving](https://mlsanity.com/paper/2604.16583/) (2026)
- [Cost-Ordered Feasibility for Multi-Armed Bandits with Cost Subsidy](https://mlsanity.com/paper/2605.07171/) (2026)
- [Multinomial Subset Routing with Sum-Max Rewards and Operational Constraints](https://mlsanity.com/paper/2608.16375/) (2026)
- [Routing Should Pay for Itself: Sparse Supervision for Economical LLM Routing](https://mlsanity.com/paper/2609.37402/) (2026)
- [CopRoute: Dependence-Aware Conformal Routing for Language Models](https://mlsanity.com/paper/2610.13276/) (2026)
- [Bayesian Learning to Select Fair Classifiers](https://mlsanity.com/paper/2610.20514/) (2026)

## Abstract (arXiv)

Efficient use of large language models (LLMs) is critical for deployment at scale: without adaptive routing, systems either overpay for strong models or risk poor performance from weaker ones. Selecting the right LLM for each query is fundamentally an online decision problem: models differ in strengths, prices fluctuate, and users value accuracy and cost differently. Yet most routers are trained offline with labels for all candidate models, an assumption that breaks in deployment, where only the outcome of the chosen model is observed. We bridge this gap with BaRP, a Bandit-feedback Routing with Preferences approach that trains under the same partial-feedback restriction as deployment, while supporting preference-tunable inference: operators can dial the performance/cost trade-off at test time without retraining. Framed as a contextual bandit over prompt features and a user preference vector, our method simulates an online feedback setting during training and adapts its routing decisions to each new prompt, rather than depending on full-information offline supervision. Comprehensive experiments show that our method consistently outperforms strong offline routers by at least 12.46% and the largest LLM by at least 2.45%, and generalizes robustly for unseen tasks.