mlsanityData to 14 Oct 2026

Reward Model Routing in Alignment

arXiv 2510.02850 ↗First posted 3 Oct 2025cs.AIcited by 4

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement learning from human or AI feedback (RLHF / RLAIF) has become the standard paradigm for aligning large language models (LLMs). However, most pipelines rely on a single reward model (RM), limiting alignment quality and risking overfitting. Recent work explores RM routing--dynamically selecting an RM from a candidate pool to exploit complementary strengths while maintaining O(1) RM calls--but existing methods suffer from cold-start and insufficient exploration. We propose BayesianRouter, a hybrid routing framework that combines offline RM strengths learning with online Bayesian selection. In the offline stage, a multi-task router is trained on preference data to estimate per-RM reliability. In the online stage, a Bayesian Thompson sampling router performs per-query RM selection, initializing RM-specific weight vectors with offline embeddings as Gaussian priors and adaptively updating their posteriors with online rewards to adapt to the evolving policy distribution. Extensive experiments on instruction-following (AlpacaEval-2, Arena-Hard, MT-Bench) and reasoning (GSM8K, MMLU) benchmarks show that BayesianRouter consistently outperforms individual RMs, RM ensembling, and existing routing methods.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 34 of its 41 references that are in the map. First posted on arXiv 3 Oct 2025.

Builds on

34 of 41 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy2 Jul
  2. 2025RewardBench 2: Advancing Reward Model Evaluation2 Jun
  3. 2025HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languages16 May
  4. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style21 Oct
  5. 2024Reinforcement Learning for LLM Post-Training: A Survey23 Jul
  6. 2024RLHF Workflow: From Reward Modeling to Online RLHF13 May
  7. 2024RewardBench: Evaluating Reward Models for Language Modeling20 Mar
  8. 2024Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemble30 Jan
  9. 2023A Survey of Reinforcement Learning from Human Feedback22 Dec
  10. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  11. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  12. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedback17 May

Other references · 22

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  6. 2024GPT-4o System Cardcited by 4,588
  7. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  8. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  9. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  10. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
  11. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 372
  12. 2024Direct Language Model Alignment from Online AI Feedbackcited by 260
10 more
  1. 2023Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Modelscited by 145
  2. 2019A Survey on Practical Applications of Multi-Armed and Contextual Banditscited by 45
  3. 2025Improve LLM-as-a-Judge Ability as a General Abilitycited by 37
  4. 2024CREAM: Consistency Regularized Self-Rewarding Language Modelscited by 35
  5. 2015A Survey on Contextual Multi-armed Banditscited by 34
  6. 2025LLM Bandit: Cost-Efficient LLM Generation via Preference-Conditioned Dynamic Routingcited by 34
  7. 2025Prompt-to-Leaderboardcited by 30
  8. 2024xFinder: Large Language Models as Automated Evaluators for Reliable Evaluationcited by 19
  9. 2024LASeR: Learning to Adaptively Select Reward Models with Multi-Armed Banditscited by 7
  10. 2025ORI: O Routing Intelligencecited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Replacing Parameters with Preferences: Federated Alignment of Heterogeneous Vision-Language Modelscited by 0
  2. 2026Replacing Parameters with Preferences: Federated Alignment of Heterogeneous Vision-Language Modelscited by 0
  3. 2026DCRL: Decoupling and Coupling Reinforcement Learning via Policy-Reward Manifold Alignmentcited by 0
  4. 2026ROUTEAUDIT: Interaction-Aware Identification for Budgeted Multi-Verifier Routingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Ask a Strong LLM Judge when Your Reward Model is Uncertain
  2. 2025The Alignment Auditor: A Bayesian Framework for Verifying and Refining LLM Objectives
  3. 2024Bayesian Reward Models for LLM Alignment
  4. 2025BOTS: A Unified Framework for Bayesian Online Task Selection in LLM Reinforcement Finetuning
  5. 2024Exploring Domain Robust Lightweight Reward Models based on Router Mechanism
  6. 2024Towards Reliable Alignment: Uncertainty-aware RLHF
  7. 2024Approximated Variational Bayesian Inverse Reinforcement Learning for Large Language Model Alignment
  8. 2025Uncertainty Quantification for Large Language Model Reward Learning under Heterogeneous Human Feedback