mlsanityData to 7 Oct 2026

Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Models

arXiv 2311.08692 ↗First posted 15 Nov 2023cs.CL cs.LGcited by 141

On the map: LLM alignment / RLHF / preference optimisation → Process reward models for reasoning

Abstract (arXiv)

The complementary potential of Large Language Models (LLM) assumes off-the-shelf LLMs have heterogeneous expertise in a wide range of domains and tasks so that an ensemble of LLMs can achieve consistently better performance. Existing ensemble methods for LLMs mainly focus on reward model ranking of outputs, leading to significant computation overhead. To combat this issue, we revisit the complementary potential of LLMs and further elaborate it by mining latent expertise with off-the-shelf reward models. We propose Zooter, a reward-guided routing method distilling rewards on training queries to train a routing function, which can precisely distribute each query to the LLM with expertise about it. We also integrate a tag-based label enhancement to mitigate noise from uncertainty when using rewards as silver supervision. Zooter shows computation efficiency in inference as it introduces only a minor computation overhead of a routing function compared with reward model ranking methods. We evaluate Zooter on a comprehensive benchmark collection with 26 subsets on different domains and tasks. Zooter outperforms the best single model on average and ranks first on 44% of tasks, even surpassing multiple reward model ranking methods.

Key facts. Cited by 141 papers in the map (data through 7 Oct 2026). Placed in Process reward models for reasoning, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 22 of its 37 references that are in the map. First posted on arXiv 15 Nov 2023.

Builds on

22 of 37 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Solving math word problems with process- and outcome-based feedback25 Nov

Other references · 21

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2023Let's Verify Step by Stepcited by 3,598
  8. 2023Qwen Technical Reportcited by 3,273
  9. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  10. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  11. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  12. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
9 more
  1. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  2. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  3. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  4. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  5. 2023Preference Ranking Optimization for Human Alignmentcited by 283
  6. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  7. 2023Large Language Model Routing with Benchmark Datasetscited by 166
  8. 2021FastMoE: A Fast Mixture-of-Expert Training Systemcited by 110
  9. 2022Uncertainty Estimation for Language Reward Modelscited by 35

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 37; 2025: 62; 2026: 42.

141 papers.

Sort:

By area:

  1. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 633
  2. 2024Mixture-of-Agents Enhances Large Language Model Capabilitiescited by 420
  3. 2024RouterBench: A Benchmark for Multi-LLM Routing Systemcited by 222
  4. 2024More Agents Is All You Needcited by 174
  5. 2025Harnessing Multiple Large Language Models: A Survey on LLM Ensemblecited by 86
  6. 2024RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Modelscited by 84
  7. 2025Universal Model Routing for Efficient LLM Inferencecited by 80
  8. 2024Towards Modular LLMs by Building and Reusing a Library of LoRAscited by 72
  9. 2024A Unified Approach to Routing and Cascading for LLMscited by 71
  10. 2024Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel Collaborationcited by 66
  11. 2024Learning to Route Among Specialized Experts for Zero-Shot Generalizationcited by 65
  12. 2025BEST-Route: Adaptive LLM Routing with Test-Time Optimal Computecited by 63
38 more
  1. 2024A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learningcited by 52
  2. 2025Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial?cited by 52
  3. 2024EmbedLLM: Learning Compact Representations of Large Language Modelscited by 46
  4. 2024What is the Role of Small Models in the LLM Era: A Surveycited by 41
  5. 2024MetaLLM: A High-performant and Cost-efficient Dynamic Framework for Wrapping LLMscited by 39
  6. 2025The Avengers: A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary Giantscited by 37
  7. 2025KABB: Knowledge-Aware Bayesian Bandits for Dynamic Expert Coordination in Multi-Agent Systemscited by 36
  8. 2025Don't Overthink It: A Survey of Efficient R1-style Large Reasoning Modelscited by 36
  9. 2024Routoo: Learning to Route to Large Language Models Effectivelycited by 34
  10. 2024Optimising Calls to Large Language Models with Uncertainty-Based Two-Tier Selectioncited by 32
  11. 2025Beyond GPT-5: Making LLMs Cheaper and Better via Performance-Efficiency Optimized Routingcited by 32
  12. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  13. 2024Smoothie: Label Free Language Model Routingcited by 31
  14. 2025CITER: Collaborative Inference for Efficient Large Language Model Decoding with Token-Level Routingcited by 29
  15. 2025R2R: Efficiently Navigating Divergent Reasoning Paths with Small-Large Model Token Routingcited by 28
  16. 2024Determine-Then-Ensemble: Necessity of Top-k Union for Large Language Model Ensemblingcited by 27
  17. 2025RewardAnything: Generalizable Principle-Following Reward Modelscited by 27
  18. 2025Route to Reason: Adaptive Routing for LLM and Reasoning Strategy Selectioncited by 23
  19. 2026SkillOrchestra: Learning to Route Agents via Skill Transfercited by 22
  20. 2025Do We Truly Need So Many Samples? Multi-LLM Repeated Sampling Efficiently Scales Test-Time Computecited by 20
  21. 2024DAM: Dynamic Adapter Merging for Continual Video QA Learningcited by 18
  22. 2024SambaNova SN40L: Scaling the AI Memory Wall with Dataflow and Composition of Expertscited by 18
  23. 2025Doing More with Less: A Survey on Routing Strategies for Resource Optimisation in Large Language Model-Based Systemscited by 18
  24. 2025Cost-Aware Contrastive Routing for LLMscited by 16
  25. 2025xRouter: Training Cost-Aware LLMs Orchestration System via Reinforcement Learningcited by 16
  26. 2026Dynamic Model Routing and Cascading for Efficient LLM Inference: A Surveycited by 16
  27. 2024Cascade-Aware Training of Language Modelscited by 15
  28. 2024Optimal Query Allocation in Extractive QA with LLMs: A Learning-to-Defer Framework with Theoretical Guaranteescited by 15
  29. 2025EllieSQL: Cost-Efficient Text-to-SQL with Complexity-Aware Routingcited by 13
  30. 2025Efficient Training-Free Online Routing for High-Volume Multi-LLM Servingcited by 13
  31. 2024LLM Chain Ensembles for Scalable and Accurate Data Annotationcited by 11
  32. 2025Life-Cycle Routing Vulnerabilities of LLM Routercited by 10
  33. 2024Efficient Dynamic Ensembling for Multiple LLM Expertscited by 9
  34. 2025Multi-LLM Collaborative Search for Complex Problem Solvingcited by 9
  35. 2025Learning Compact Representations of LLM Abilities via Item Response Theorycited by 9
  36. 2024Hit the Sweet Spot! Span-Level Ensemble for Large Language Modelscited by 8
  37. 2025Ensemble Learning for Large Language Models in Text and Code Generation: A Surveycited by 8
  38. 2025MESS+: Dynamically Learned Inference-Time LLM Routing in Model Zoos with Service Level Guaranteescited by 8

50 most cited of 141 All 141 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Cross-Entropy Guided Routing in Mixture-of-Experts Large Language Models
  2. 2024DMoERM: Recipes of Mixture-of-Experts for Effective Reward Modeling
  3. 2025Adaptive LLM Routing under Budget Constraints
  4. 2025Language Ranker: A Lightweight Ranking framework for LLM Decoding
  5. 2025Mixture of Thoughts: Learning to Aggregate What Experts Think, Not Just What They Say
  6. 2025LLM Bandit: Cost-Efficient LLM Generation via Preference-Conditioned Dynamic Routing
  7. 2025Bayesian Mixture-of-Experts: Towards Making LLMs Know What They Don't Know
  8. 2024A Closer Look into Mixture-of-Experts in Large Language Models