# MixLLM: Dynamic Routing in Mixed Large Language Models

arXiv: https://arxiv.org/abs/2502.18482 · HTML: https://mlsanity.com/paper/2502.18482/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 36 references that are in the map.
- First posted on arXiv 9 Feb 2025.

## Builds on (most cited first, 19 of 19)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [RouteLLM: Learning to Route LLMs with Preference Data](https://mlsanity.com/paper/2406.18665/) (2024)
- [Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing](https://mlsanity.com/paper/2404.14618/) (2024)
- [Large Language Model Routing with Benchmark Datasets](https://mlsanity.com/paper/2309.15789/) (2023)
- [Observational Scaling Laws and the Predictability of Language Model Performance](https://mlsanity.com/paper/2405.10938/) (2024)
- [MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmark](https://mlsanity.com/paper/2405.12209/) (2024)
- [Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modeling](https://mlsanity.com/paper/2308.06077/) (2023)
- [Empowering Time Series Analysis with Large Language Models: A Survey](https://mlsanity.com/paper/2402.03182/) (2024)
- [MetaLLM: A High-performant and Cost-efficient Dynamic Framework for Wrapping LLMs](https://mlsanity.com/paper/2407.10834/) (2024)
- [Evolutionary Large Language Model for Automated Feature Transformation](https://mlsanity.com/paper/2405.16203/) (2024)
- [LLM-Enhanced User-Item Interactions: Leveraging Edge Information for Optimized Recommendations](https://mlsanity.com/paper/2402.09617/) (2024)
- [Multi-facet Contextual Bandits: A Neural Network Perspective](https://mlsanity.com/paper/2106.03039/) (2021)
- [Reinforcement Feature Transformation for Polymer Property Performance Prediction](https://mlsanity.com/paper/2409.15616/) (2024)
- [Revolutionizing Biomarker Discovery: Leveraging Generative AI for Bio-Knowledge-Embedded Continuous Space Exploration](https://mlsanity.com/paper/2409.15612/) (2024)

## Built on by (most cited first, 30 of 59)

- [Harnessing Multiple Large Language Models: A Survey on LLM Ensemble](https://mlsanity.com/paper/2502.18036/) (2025)
- [Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Models](https://mlsanity.com/paper/2503.24377/) (2025)
- [R2R: Efficiently Navigating Divergent Reasoning Paths with Small-Large Model Token Routing](https://mlsanity.com/paper/2505.21600/) (2025)
- [Route to Reason: Adaptive Routing for LLM and Reasoning Strategy Selection](https://mlsanity.com/paper/2505.19435/) (2025)
- [RouterArena: An Open Platform for Comprehensive Comparison of LLM Routers](https://mlsanity.com/paper/2510.00202/) (2025)
- [A Survey of Slow Thinking-based Reasoning LLMs using Reinforced Learning and Inference-time Scaling Law](https://mlsanity.com/paper/2505.02665/) (2025)
- [Online Multi-LLM Selection via Contextual Bandits under Unstructured Context Evolution](https://mlsanity.com/paper/2506.17670/) (2025)
- [Dynamic Model Routing and Cascading for Efficient LLM Inference: A Survey](https://mlsanity.com/paper/2603.04445/) (2026)
- [GlimpRouter: Efficient Collaborative Inference by Glimpsing One Token of Thoughts](https://mlsanity.com/paper/2601.05110/) (2026)
- [EvolveRouter: Co-Evolving Routing and Prompt for Multi-Agent Question Answering](https://mlsanity.com/paper/2604.05149/) (2026)
- [A Survey on Collaborative Mechanisms Between Large and Small Language Models](https://mlsanity.com/paper/2505.07460/) (2025)
- [Efficient Training-Free Online Routing for High-Volume Multi-LLM Serving](https://mlsanity.com/paper/2509.02718/) (2025)
- [Energy Use of AI Inference, Efficiency Pathways, and Test-Time Scaling](https://mlsanity.com/paper/2509.20241/) (2025)
- [AIBuildAI: An AI Agent for Automatically Building AI Models](https://mlsanity.com/paper/2604.14455/) (2026)
- [Unsupervised Feature Transformation via In-context Generation, Generator-critic LLM Agents, and Duet-play Teaming](https://mlsanity.com/paper/2504.21304/) (2025)
- [Controlling Thinking Speed in Reasoning Models](https://mlsanity.com/paper/2507.03704/) (2025)
- [Learning Compact Representations of LLM Abilities via Item Response Theory](https://mlsanity.com/paper/2510.00844/) (2025)
- [When Routing Collapses: On the Degenerate Convergence of LLM Routers](https://mlsanity.com/paper/2602.03478/) (2026)
- [Budget-Aware Agentic Routing via Boundary-Guided Training](https://mlsanity.com/paper/2602.21227/) (2026)
- [Efficient Post-Training Refinement of Latent Reasoning in Large Language Models](https://mlsanity.com/paper/2506.08552/) (2025)
- [Teaching Time Series to See and Speak: Forecasting with Aligned Visual and Textual Perspectives](https://mlsanity.com/paper/2506.24124/) (2025)
- [LLM Routing with Dueling Feedback](https://mlsanity.com/paper/2510.00841/) (2025)
- [Fast or Better? Balancing Accuracy and Cost in Retrieval-Augmented Generation with Flexible User Control](https://mlsanity.com/paper/2502.12145/) (2025)
- [Effective LoRA Adapter Routing using Task Representations](https://mlsanity.com/paper/2601.21795/) (2026)
- [DiSRouter: Distributed Self-Routing for LLM Selections](https://mlsanity.com/paper/2510.19208/) (2025)
- [Dynamic Quality-Latency Aware Routing for LLM Inference in Wireless Edge-Device Networks](https://mlsanity.com/paper/2508.11291/) (2025)
- [Reference-Free Rating of LLM Responses via Latent Information](https://mlsanity.com/paper/2509.24678/) (2025)
- [Chimera: Latency- and Performance-Aware Multi-agent Serving for Heterogeneous LLMs](https://mlsanity.com/paper/2603.22206/) (2026)
- [ParetoBandit: Budget-Paced Adaptive Routing for Non-Stationary LLM Serving](https://mlsanity.com/paper/2604.00136/) (2026)
- [LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers](https://mlsanity.com/paper/2608.06867/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) exhibit potential artificial generic intelligence recently, however, their usage is costly with high response latency. Given mixed LLMs with their own strengths and weaknesses, LLM routing aims to identify the most suitable model for each query in the stream to maximize response quality and minimize cost and latency. However, the challenges involve: (1) dynamic trade-offs among quality, cost, and latency; (2) enabling continual learning in deployed systems; and (3) navigating a varying (e.g., new LLM addition or old LLM removal) set of LLM candidates over time. To bridge these gaps, we develop MixLLM, a dynamic contextual-bandit-based routing system for query-LLM assignment. Specifically, we first leverage query tags to enhance query embeddings for the routing task. Next, we design lightweight prediction models to estimate the response qualities and costs of queries over LLMs. We then devise a meta-decision maker to choose the query-LLM assignments to best tradeoff response quality, cost, and latency. Finally, the system benefits from continual training, allowing it to adapt to evolving queries and user feedback over time. Our extensive experiments show that MixLLM achieves the best trade-offs in response quality, cost, and latency (97.25% of GPT-4's quality at 24.18% of the cost under the time constraint).