mlsanityData to 7 Oct 2026

Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing

arXiv 2404.14618 ↗First posted 22 Apr 2024cs.LG cs.AI cs.CLcited by 354

Not on the map. Why?

Cited by 66 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LLMs) excel in most NLP tasks but also require expensive cloud servers for deployment due to their size, while smaller models that can be deployed on lower cost (e.g., edge) devices, tend to lag behind in terms of response quality. Therefore in this work we propose a hybrid inference approach which combines their respective strengths to save cost and maintain quality. Our approach uses a router that assigns queries to the small or large model based on the predicted query difficulty and the desired quality level. The desired quality level can be tuned dynamically at test time to seamlessly trade quality for cost as per the scenario requirements. In experiments our approach allows us to make up to 40% fewer calls to the large model, with no drop in response quality.

Key facts. Cited by 354 papers in the map (data through 7 Oct 2026). Builds on 27 of its 45 references that are in the map. First posted on arXiv 22 Apr 2024.

Builds on

27 of 45 references.

References · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  8. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  9. 2023A Survey of Large Language Modelscited by 2,537
  10. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  11. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  12. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
15 more
  1. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  2. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  3. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  4. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  5. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  6. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  7. 2020Class-incremental learning: survey and performance evaluation on image classificationcited by 446
  8. 2023Speculative Decoding with Big Little Decodercited by 163
  9. 2021Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Bettercited by 153
  10. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  11. 2019Fast Structured Decoding for Sequence Modelscited by 59
  12. 2019Video Object Segmentation and Tracking: A Surveycited by 49
  13. 2022Scaling up Trustless DNN Inference with Zero-Knowledge Proofscited by 30
  14. 2022A global analysis of metrics used for measuring performance in natural language processingcited by 17
  15. 2022On Efficient Approximate Queries over Machine Learning Modelscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 41; 2025: 111; 2026: 202.

354 papers.

Sort:

By area:

  1. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 633
  2. 2024Token-Budget-Aware LLM Reasoningcited by 242
  3. 2024GraphRouter: A Graph-based Router for LLM Selectionscited by 154
  4. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  5. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 106
  6. 2024A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challengescited by 86
  7. 2025Harnessing Multiple Large Language Models: A Survey on LLM Ensemblecited by 86
  8. 2024RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Modelscited by 84
  9. 2025Universal Model Routing for Efficient LLM Inferencecited by 80
  10. 2025MasRouter: Learning to Route LLMs for Multi-Agent Systemscited by 80
  11. 2025ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learningcited by 78
  12. 2024A Unified Approach to Routing and Cascading for LLMscited by 71
38 more
  1. 2024Learning to Route LLMs with Confidence Tokenscited by 66
  2. 2025Router-R1: Teaching LLMs Multi-Round Routing and Aggregation via Reinforcement Learningcited by 64
  3. 2025BEST-Route: Adaptive LLM Routing with Test-Time Optimal Computecited by 63
  4. 2024TensorOpera Router: A Multi-Model Router for Efficient LLM Inferencecited by 59
  5. 2025RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMscited by 58
  6. 2025MixLLM: Dynamic Routing in Mixed Large Language Modelscited by 56
  7. 2024Game-theoretic LLM: Agent Workflow for Negotiation Gamescited by 55
  8. 2024Harnessing the Power of Multiple Minds: Lessons Learned from LLM Routingcited by 43
  9. 2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 43
  10. 2024What is the Role of Small Models in the LLM Era: A Surveycited by 41
  11. 2025IRT-Router: Effective and Interpretable Multi-LLM Routing via Item Response Theorycited by 40
  12. 2024MetaLLM: A High-performant and Cost-efficient Dynamic Framework for Wrapping LLMscited by 39
  13. 2024Faster Cascades via Speculative Decodingcited by 38
  14. 2024Cost-Effective Online Multi-LLM Selection with Versatile Reward Modelscited by 36
  15. 2025Don't Overthink It: A Survey of Efficient R1-style Large Reasoning Modelscited by 36
  16. 2025OmniRouter: Budget and Performance Controllable Multi-LLM Routingcited by 35
  17. 2025Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Modelscited by 35
  18. 2026LLMRouterBench: A Massive Benchmark and Unified Framework for LLM Routingcited by 35
  19. 2024Optimising Calls to Large Language Models with Uncertainty-Based Two-Tier Selectioncited by 32
  20. 2025Internet of Agents: Fundamentals, Applications, and Challengescited by 31
  21. 2025Prompt-to-Leaderboardcited by 27
  22. 2025When One LLM Drools, Multi-LLM Collaboration Rulescited by 25
  23. 2025Capability Instruction Tuning: A New Paradigm for Dynamic LLM Routingcited by 25
  24. 2024CLAVE: An Adaptive Framework for Evaluating Values of LLM Generated Responsescited by 23
  25. 2025Route to Reason: Adaptive Routing for LLM and Reasoning Strategy Selectioncited by 23
  26. 2025FastCache: Fast Caching for Diffusion Transformer Through Learnable Linear Approximationcited by 23
  27. 2024HyPA-RAG: A Hybrid Parameter Adaptive Retrieval-Augmented Generation System for AI Legal and Policy Applicationscited by 22
  28. 2026SkillOrchestra: Learning to Route Agents via Skill Transfercited by 22
  29. 2025Adaptive LLM Routing under Budget Constraintscited by 21
  30. 2024BudgetMLAgent: A Cost-Effective LLM Multi-Agent system for Automating Machine Learning Taskscited by 19
  31. 2025Doing More with Less: A Survey on Routing Strategies for Resource Optimisation in Large Language Model-Based Systemscited by 18
  32. 2025Online Multi-LLM Selection via Contextual Bandits under Unstructured Context Evolutioncited by 18
  33. 2025Towards Concise and Adaptive Thinking in Large Reasoning Models: A Surveycited by 18
  34. 2024Agreement-Based Cascading for Efficient Inferencecited by 16
  35. 2025Cost-Aware Contrastive Routing for LLMscited by 16
  36. 2025xRouter: Training Cost-Aware LLMs Orchestration System via Reinforcement Learningcited by 16
  37. 2026GlimpRouter: Efficient Collaborative Inference by Glimpsing One Token of Thoughtscited by 16
  38. 2026Dynamic Model Routing and Cascading for Efficient LLM Inference: A Surveycited by 16

50 most cited of 354 All 354 (JSON)