Data to 7 Oct 2026

Reward Model Ensembles Help Mitigate Overoptimization

arXiv 2310.02743 ↗First posted 4 Oct 2023cs.LGcited by 225

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement learning from human feedback (RLHF) is a standard approach for fine-tuning large language models to follow instructions. As part of this process, learned reward models are used to approximately model human preferences. However, as imperfect representations of the "true" reward, these learned reward models are susceptible to overoptimization. Gao et al. (2023) studied this phenomenon in a synthetic human feedback setup with a significantly larger "gold" reward model acting as the true reward (instead of humans) and showed that overoptimization remains a persistent problem regardless of the size of the proxy reward model and training data used. Using a similar setup, we conduct a systematic study to evaluate the efficacy of using ensemble-based conservative optimization objectives, specifically worst-case optimization (WCO) and uncertainty-weighted optimization (UWO), for mitigating reward model overoptimization when using two optimization methods: (a) best-of-n sampling (BoN) (b) proximal policy optimization (PPO). We additionally extend the setup of Gao et al. (2023) to include 25% label noise to better mirror real-world conditions. Both with and without label noise, we find that conservative optimization practically eliminates overoptimization and improves performance by up to 70% for BoN sampling. For PPO, ensemble-based conservative optimization always reduces overoptimization and outperforms single reward model optimization. Moreover, combining it with a small KL penalty successfully prevents overoptimization at no performance cost. Overall, our results demonstrate that ensemble-based conservative optimization can effectively counter overoptimization.

Key facts. Cited by 225 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 43 of its 55 references that are in the map. First posted on arXiv 4 Oct 2023.

Builds on

43 of 55 references.

Closest ancestors · 17

Same direction, newest first

5 more

Other references · 26

2022Training language models to follow instructions with human feedbackcited by 15,712
2017Proximal Policy Optimization Algorithmscited by 14,441
2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
2023Let's Verify Step by Stepcited by 3,598
2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
2020Learning to summarize from human feedbackcited by 2,562
2021Convex optimizationcited by 2,479
2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
2022Solving math word problems with process- and outcome-based feedbackcited by 871
2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748

14 more

2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
2020MOPO: Model-based Offline Policy Optimizationcited by 630
2022Self-critiquing models for assisting human evaluatorscited by 372
2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
2021Recursively Summarizing Books with Human Feedbackcited by 287
2019Self-Supervised Exploration via Disagreementcited by 263
2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learningcited by 125
2023Training Language Models with Language Feedback at Scalecited by 125
2018Model-Based Active Explorationcited by 105
2021Consequences of Misaligned AIcited by 79
2018The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communitiescited by 78
2016Learning and Policy Search in Stochastic Dynamical Systems with Bayesian Neural Networkscited by 49
2019Explicit Explore-Exploit Algorithms in Continuous State Spacescited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 77; 2025: 79; 2026: 64.

225 papers.

Sort:

By area:

2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 195
2024Disentangling Length from Quality in Direct Preference Optimizationcited by 194
2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
2025Scalable Best-of-N Selection for Large Language Models via Self-Certaintycited by 182
2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 126
2024Theoretical guarantees on the best-of-n alignment policycited by 116
2024Fast Best-of-N Decoding via Speculative Rejectioncited by 112

38 more

2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
2024Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedbackcited by 107
2024Human Alignment of Large Language Models through Online Preference Optimisationcited by 90
2025Test-time Alignment of Diffusion Models without Reward Over-optimizationcited by 89
2025Reward Shaping to Mitigate Reward Hacking in RLHFcited by 88
2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
2024Dense Reward for Free in Reinforcement Learning from Human Feedbackcited by 77
2024InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modelingcited by 76
2024RRM: Robust Reward Model Training Mitigates Reward Hackingcited by 75
2024Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknowncited by 71
2024Self-Generated Critiques Boost Reward Modeling for Language Modelscited by 69
2024Robust Preference Optimization through Reward Model Distillationcited by 67
2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66
2024TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Samplingcited by 61
2025Multi-Agent Verification: Scaling Test-Time Compute with Multiple Verifierscited by 58
2024Learn Your Reference Model for Real Good Alignmentcited by 56
2023Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensemblescited by 55
2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
2024Online Iterative Reinforcement Learning from Human Feedback with General Preference Modelcited by 52
2025Exploring the Limit of Outcome Reward for Learning Mathematical Reasoningcited by 51
2024Correlated Proxies: A New Definition and Improved Mitigation for Reward Hackingcited by 50
2025Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systemscited by 49
2024Offline Regularised Reinforcement Learning for Large Language Models Alignmentcited by 48
2024Rethinking Bradley-Terry Models in Preference-Based Reward Modeling: Foundations, Theory, and Alternativescited by 48
2025GRPO-Guard: Mitigating Implicit Over-Optimization in Flow Matching via Regulated Clippingcited by 48
2024Improving Reward Models with Synthetic Critiquescited by 44
2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 43
2023The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedbackcited by 42
2025Beyond Reward Hacking: Causal Rewards for Large Language Model Alignmentcited by 42
2024Bayesian Reward Models for LLM Alignmentcited by 35
2024CREAM: Consistency Regularized Self-Rewarding Language Modelscited by 35
2024InfAlign: Inference-aware language model alignmentcited by 34
2024Overcoming Reward Overoptimization via Adversarial Policy Optimization with Lightweight Uncertainty Estimationcited by 33
2024Scalable Ranked Preference Optimization for Text-to-Image Generationcited by 33
2024MusicRL: Aligning Music Generation to Human Preferencescited by 32
2024Reward-Robust RLHF in LLMscited by 32
2025Inference-Time Reward Hacking in Large Language Modelscited by 32
2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 30

50 most cited of 225 All 225 (JSON)

Similar, not cited

Close in wording, no citation either way