mlsanityData to 7 Oct 2026

Scalable Ensembling For Mitigating Reward Overoptimisation

arXiv 2406.01013 ↗First posted 3 Jun 2024cs.LG cs.CLcited by 14

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) has enabled significant advancements within language modeling for powerful, instruction-following models. However, the alignment of these models remains a pressing challenge as the policy tends to overfit the learned ``proxy" reward model past an inflection point of utility as measured by a ``gold" reward model that is more performant -- a phenomenon known as overoptimisation. Prior work has mitigated this issue by computing a pessimistic statistic over an ensemble of reward models, which is common in Offline Reinforcement Learning but incredibly costly for language models with high memory requirements, making such approaches infeasible for sufficiently large models. To this end, we propose using a shared encoder but separate linear heads. We find this leads to similar performance as the full ensemble while allowing tremendous savings in memory and time required for training for models of similar size.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 13 of its 24 references that are in the map. First posted on arXiv 3 Jun 2024.

Builds on

13 of 24 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Overcoming Reward Overoptimization via Adversarial Policy Optimization with Lightweight Uncertainty Estimation8 Mar
  2. 2024Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemble30 Jan
  3. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking14 Dec
  4. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  5. 2022Scaling Laws for Reward Model Overoptimization19 Oct

Other references · 8

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2022Emergent Abilities of Large Language Modelscited by 2,483
  4. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  5. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  6. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  7. 2023An Overview of Catastrophic AI Riskscited by 214
  8. 2019On the Feasibility of Learning, Rather than Assuming, Human Biases for Reward Inferencecited by 49

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 10; 2026: 2.

14 papers.

Sort:

By area:

  1. 2025Reward Shaping to Mitigate Reward Hacking in RLHFcited by 88
  2. 2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 43
  3. 2025Inference-Time Reward Hacking in Large Language Modelscited by 32
  4. 2024Test-Time Alignment via Hypothesis Reweightingcited by 11
  5. 2025Rethinking Diverse Human Preference Learning through Principal Component Analysiscited by 10
  6. 2025Ensembles of Low-Rank Expert Adapterscited by 9
  7. 2025Ensemble Learning for Large Language Models in Text and Code Generation: A Surveycited by 8
  8. 2025Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunitiescited by 7
  9. 2025Reusing Embeddings: Reproducible Reward Model Research in Large Language Model Alignment without GPUscited by 4
  10. 2026Reinforcement Learning from Human Feedback: A Statistical Perspectivecited by 4
  11. 2025RIVAL: Reinforcement Learning with Iterative and Adversarial Optimization for Machine Translationcited by 3
  12. 2025Limited Preference Data? Learning Better Reward Model with Latent Space Synthesiscited by 2
2 more
  1. 2025UMM-RM: An Upcycle-and-Merge MoE Reward Model for Mitigating Reward Hackingcited by 0
  2. 2026Theoretical Limits of Language Model Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Off-Policy Corrected Reward Modeling for Reinforcement Learning from Human Feedback
  2. 2023Reward Model Ensembles Help Mitigate Overoptimization
  3. 2025Reward Model Overoptimisation in Iterated RLHF
  4. 2025Generalist Reward Models: Found Inside Large Language Models
  5. 2023Confronting Reward Model Overoptimization with Constrained RLHF
  6. 2026Real-Time Aligned Reward Model beyond Semantics
  7. 2024Dense Reward for Free in Reinforcement Learning from Human Feedback
  8. 2024Navigating Noisy Feedback: Enhancing Reinforcement Learning with Error-Prone Language Models