mlsanityData to 7 Oct 2026

MixLLM: Dynamic Routing in Mixed Large Language Models

arXiv 2502.18482 ↗First posted 9 Feb 2025cs.CL cs.AI cs.DBcited by 56

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) exhibit potential artificial generic intelligence recently, however, their usage is costly with high response latency. Given mixed LLMs with their own strengths and weaknesses, LLM routing aims to identify the most suitable model for each query in the stream to maximize response quality and minimize cost and latency. However, the challenges involve: (1) dynamic trade-offs among quality, cost, and latency; (2) enabling continual learning in deployed systems; and (3) navigating a varying (e.g., new LLM addition or old LLM removal) set of LLM candidates over time. To bridge these gaps, we develop MixLLM, a dynamic contextual-bandit-based routing system for query-LLM assignment. Specifically, we first leverage query tags to enhance query embeddings for the routing task. Next, we design lightweight prediction models to estimate the response qualities and costs of queries over LLMs. We then devise a meta-decision maker to choose the query-LLM assignments to best tradeoff response quality, cost, and latency. Finally, the system benefits from continual training, allowing it to adapt to evolving queries and user feedback over time. Our extensive experiments show that MixLLM achieves the best trade-offs in response quality, cost, and latency (97.25% of GPT-4's quality at 24.18% of the cost under the time constraint).

Key facts. Cited by 56 papers in the map (data through 7 Oct 2026). Builds on 19 of its 36 references that are in the map. First posted on arXiv 9 Feb 2025.

Builds on

19 of 36 references.

References · 19

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 633
  8. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 354
  9. 2023Large Language Model Routing with Benchmark Datasetscited by 166
  10. 2024Observational Scaling Laws and the Predictability of Language Model Performancecited by 128
  11. 2024MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmarkcited by 103
  12. 2023Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modelingcited by 94
7 more
  1. 2024Empowering Time Series Analysis with Large Language Models: A Surveycited by 60
  2. 2024MetaLLM: A High-performant and Cost-efficient Dynamic Framework for Wrapping LLMscited by 39
  3. 2024Evolutionary Large Language Model for Automated Feature Transformationcited by 29
  4. 2024LLM-Enhanced User-Item Interactions: Leveraging Edge Information for Optimized Recommendationscited by 24
  5. 2021Multi-facet Contextual Bandits: A Neural Network Perspectivecited by 20
  6. 2024Reinforcement Feature Transformation for Polymer Property Performance Predictioncited by 13
  7. 2024Revolutionizing Biomarker Discovery: Leveraging Generative AI for Bio-Knowledge-Embedded Continuous Space Explorationcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 25; 2026: 31.

56 papers.

Sort:

By area:

  1. 2025Harnessing Multiple Large Language Models: A Survey on LLM Ensemblecited by 86
  2. 2025Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Modelscited by 35
  3. 2025R2R: Efficiently Navigating Divergent Reasoning Paths with Small-Large Model Token Routingcited by 28
  4. 2025Route to Reason: Adaptive Routing for LLM and Reasoning Strategy Selectioncited by 23
  5. 2025RouterArena: An Open Platform for Comprehensive Comparison of LLM Routerscited by 19
  6. 2025A Survey of Slow Thinking-based Reasoning LLMs using Reinforced Learning and Inference-time Scaling Lawcited by 18
  7. 2025Online Multi-LLM Selection via Contextual Bandits under Unstructured Context Evolutioncited by 18
  8. 2026GlimpRouter: Efficient Collaborative Inference by Glimpsing One Token of Thoughtscited by 16
  9. 2026Dynamic Model Routing and Cascading for Efficient LLM Inference: A Surveycited by 16
  10. 2025Efficient Training-Free Online Routing for High-Volume Multi-LLM Servingcited by 13
  11. 2026EvolveRouter: Co-Evolving Routing and Prompt for Multi-Agent Question Answeringcited by 13
  12. 2025A Survey on Collaborative Mechanisms Between Large and Small Language Modelscited by 12
38 more
  1. 2025Energy Use of AI Inference, Efficiency Pathways, and Test-Time Scalingcited by 12
  2. 2025Unsupervised Feature Transformation via In-context Generation, Generator-critic LLM Agents, and Duet-play Teamingcited by 11
  3. 2025Controlling Thinking Speed in Reasoning Modelscited by 10
  4. 2026When Routing Collapses: On the Degenerate Convergence of LLM Routerscited by 10
  5. 2026AIBuildAI: An AI Agent for Automatically Building AI Modelscited by 10
  6. 2025Learning Compact Representations of LLM Abilities via Item Response Theorycited by 9
  7. 2026Budget-Aware Agentic Routing via Boundary-Guided Trainingcited by 8
  8. 2025Efficient Post-Training Refinement of Latent Reasoning in Large Language Modelscited by 7
  9. 2025Teaching Time Series to See and Speak: Forecasting with Aligned Visual and Textual Perspectivescited by 7
  10. 2025Fast or Better? Balancing Accuracy and Cost in Retrieval-Augmented Generation with Flexible User Controlcited by 6
  11. 2025LLM Routing with Dueling Feedbackcited by 6
  12. 2026Effective LoRA Adapter Routing using Task Representationscited by 6
  13. 2025DiSRouter: Distributed Self-Routing for LLM Selectionscited by 5
  14. 2025Dynamic Quality-Latency Aware Routing for LLM Inference in Wireless Edge-Device Networkscited by 4
  15. 2025Reference-Free Rating of LLM Responses via Latent Informationcited by 4
  16. 2026Chimera: Latency- and Performance-Aware Multi-agent Serving for Heterogeneous LLMscited by 4
  17. 2025Near-Optimal Online Deployment and Routing for Streaming LLMscited by 3
  18. 2026ParetoBandit: Budget-Paced Adaptive Routing for Non-Stationary LLM Servingcited by 3
  19. 2026Opportunity Is Not Realizability: Selection-Valid Diagnostics for Multi-LLM Routingcited by 3
  20. 2025Balancing Information Accuracy and Response Timeliness in Networked LLMscited by 2
  21. 2026SeqRoute: Global Budget-Aware Sequential LLM Routing via Offline Reinforcement Learningcited by 2
  22. 2026WISERouter: LLM Routing with Workload Budget Constraintcited by 2
  23. 2025Uncertainty-Aware Collaborative System of Large and Small Models for Multimodal Sentiment Analysiscited by 1
  24. 2025CONCUR: A Framework for Continual Constrained and Unconstrained Routingcited by 1
  25. 2026Optimal Splitting of Language Models from Mixtures to Specialized Domainscited by 1
  26. 2026ConsRoute:Consistency-Aware Adaptive Query Routing for Cloud-Edge-Device Large Language Modelscited by 1
  27. 2026The Workload-Router-Pool Architecture for LLM Inference Optimization: A Vision Paper from the vLLM Semantic Router Projectcited by 1
  28. 2026RouteJudge: An Open Platform for Reproducible and Preference-Aware LLM Routingcited by 1
  29. 2026Linguistic Firewall: Geometry as Defense in Multi-Agent Systems Routingcited by 1
  30. 2026SCATE: Learning to Supervise Coding Agents for Cost-Effective Test Generationcited by 1
  31. 2026PyroDash: Cost-Efficient Token-Level Small-Large Language Model Collaborative Inferencecited by 1
  32. 2026TRACE-ROUTER: Task-Consistent and Adaptive Online Routing for Agentic AIcited by 1
  33. 2026Pre-Inference Routing for Cost-Efficient Document Field Extractioncited by 1
  34. 2026LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routerscited by 1
  35. 2025CoE-Ops: Collaboration of LLM-based Experts for AIOps Question-Answeringcited by 0
  36. 2026Evolving Demonstration Optimization for Chain-of-Thought Feature Transformationcited by 0
  37. 2026MemBoost: A Memory-Boosted Framework for Cost-Aware LLM Inferencecited by 0
  38. 2026Capturing LLM Capabilities via Evidence-Calibrated Query Clusteringcited by 0

50 most cited of 56 All 56 (JSON)