mlsanityData to 7 Oct 2026

Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemble

arXiv 2401.16635 ↗First posted 30 Jan 2024cs.LG cs.AI cs.CLcited by 43

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) is a widely adopted approach for aligning large language models with human values. However, RLHF relies on a reward model that is trained with a limited amount of human preference data, which could lead to inaccurate predictions. As a result, RLHF may produce outputs that are misaligned with human values. To mitigate this issue, we contribute a reward ensemble method that allows the reward model to make more accurate predictions. As using an ensemble of large language model-based reward models can be computationally and resource-expensive, we explore efficient ensemble methods including linear-layer ensemble and LoRA-based ensemble. Empirically, we run Best-of-n and Proximal Policy Optimization with our ensembled reward models, and verify that our ensemble methods help improve the alignment performance of RLHF outputs.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 24 of its 32 references that are in the map. First posted on arXiv 30 Jan 2024.

Builds on

24 of 32 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024WARM: On the Benefits of Weight Averaged Reward Models22 Jan
  2. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking14 Dec
  3. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  4. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Training2 Jun
  5. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  6. 2022Uncertainty Estimation for Language Reward Models14 Mar
  7. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 17

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  7. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  8. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  9. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  10. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  11. 2016Concrete Problems in AI Safetycited by 1,582
  12. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
5 more
  1. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  2. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  3. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 496
  4. 2017AI Safety Gridworldscited by 108
  5. 2023LoRA ensembles for large language model fine-tuningcited by 65

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 18; 2025: 19; 2026: 6.

43 papers.

Sort:

By area:

  1. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
  2. 2025Reward Shaping to Mitigate Reward Hacking in RLHFcited by 88
  3. 2025What Makes a Reward Model a Good Teacher? An Optimization Perspectivecited by 70
  4. 2024A Survey on LoRA of Large Language Modelscited by 67
  5. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  6. 2024Offline Regularised Reinforcement Learning for Large Language Models Alignmentcited by 48
  7. 2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 43
  8. 2024Bayesian Reward Models for LLM Alignmentcited by 35
  9. 2024CREAM: Consistency Regularized Self-Rewarding Language Modelscited by 35
  10. 2024Reward-Robust RLHF in LLMscited by 32
  11. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 30
  12. 2024Post-hoc Reward Calibration: A Case Study on Length Biascited by 29
31 more
  1. 2024On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimizationcited by 26
  2. 2024Policy Filtration for RLHF to Mitigate Noise in Reward Modelscited by 21
  3. 2025CounselBench: A Large-Scale Expert Evaluation and Adversarial Benchmarking of Large Language Models in Mental Health Question Answeringcited by 20
  4. 2025Scaling Autonomous Agents via Automatic Reward Modeling And Planningcited by 19
  5. 2024Towards a Unified View of Preference Learning for Large Language Models: A Surveycited by 17
  6. 2024Scalable Ensembling For Mitigating Reward Overoptimisationcited by 14
  7. 2024ALaRM: Align Language Models via Hierarchical Rewards Modelingcited by 11
  8. 2024Test-Time Alignment via Hypothesis Reweightingcited by 11
  9. 2025Bradley-Terry and Multi-Objective Reward Modeling Are Complementarycited by 10
  10. 2025Ensembles of Low-Rank Expert Adapterscited by 9
  11. 2024Effective Generative AI: The Human-Algorithm Centaurcited by 8
  12. 2025Ensemble Learning for Large Language Models in Text and Code Generation: A Surveycited by 8
  13. 2024LASeR: Learning to Adaptively Select Reward Models with Multi-Armed Banditscited by 7
  14. 2024Towards Reliable Alignment: Uncertainty-aware RLHFcited by 7
  15. 2025Off-Policy Corrected Reward Modeling for Reinforcement Learning from Human Feedbackcited by 6
  16. 2024Towards Comprehensive Preference Data Collection for Reward Modelingcited by 5
  17. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 5
  18. 2025Energy-Based Reward Models for Robust Language Model Alignmentcited by 4
  19. 2025Toward Evaluative Thinking: Meta Policy Optimization with Evolving Reward Modelscited by 4
  20. 2025Modeling and Optimizing User Preferences in AI Copilots: A Comprehensive Survey and Taxonomycited by 3
  21. 2025Reward Model Routing in Alignmentcited by 3
  22. 2025When Human Preferences Flip: An Instance-Dependent Robust Loss for RLHFcited by 3
  23. 2026RewardUQ: A Unified Framework for Uncertainty-Aware Reward Modelscited by 3
  24. 2025Industrial LLM-based Code Optimization under Regulation: A Mixture-of-Agents Approachcited by 2
  25. 2025Explainable reinforcement learning from human feedback to improve alignmentcited by 1
  26. 2025Towards Reliable, Uncertainty-Aware Alignmentcited by 0
  27. 2025ACPO: Adaptive Curriculum Policy Optimization for Aligning Vision-Language Models in Complex Reasoningcited by 0
  28. 2025UMM-RM: An Upcycle-and-Merge MoE Reward Model for Mitigating Reward Hackingcited by 0
  29. 2026Theoretical Limits of Language Model Alignmentcited by 0
  30. 2026SCOPE: Synthetic Conditional Objectives for Policy Evolution in Black-Box Combinatorial Optimizationcited by 0
  31. 2026PHIRL: Aligning Learned Rewards with Task Progress for Inverse Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling
  2. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  3. 2024Solving the Inverse Alignment Problem for Efficient RLHF
  4. 2023Reward Model Ensembles Help Mitigate Overoptimization
  5. 2023Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensembles
  6. 2025Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuning
  7. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning
  8. 2024Optimal Design for Reward Modeling in RLHF