mlsanityData to 7 Oct 2026

Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking

arXiv 2312.09244 ↗First posted 14 Dec 2023cs.LGcited by 195

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reward models play a key role in aligning language model applications towards human preferences. However, this setup creates an incentive for the language model to exploit errors in the reward model to achieve high estimated reward, a phenomenon often termed reward hacking. A natural mitigation is to train an ensemble of reward models, aggregating over model outputs to obtain a more robust reward estimate. We explore the application of reward ensembles to alignment at both training time (through reinforcement learning) and inference time (through reranking). First, we show that reward models are underspecified: reward models that perform similarly in-distribution can yield very different rewards when used in alignment, due to distribution shift. Second, underspecification results in overoptimization, where alignment to one reward model does not improve reward as measured by another reward model trained on the same data. Third, overoptimization is mitigated by the use of reward ensembles, and ensembles that vary by their pretraining seeds lead to better generalization than ensembles that differ only by their fine-tuning seeds, with both outperforming individual reward models. However, even pretrain reward ensembles do not eliminate reward hacking: we show several qualitative reward hacking phenomena that are not mitigated by ensembling because all reward models in the ensemble exhibit similar error patterns.

Key facts. Cited by 195 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 30 of its 37 references that are in the map. First posted on arXiv 14 Dec 2023.

Builds on

30 of 37 references.

Closest ancestors · 10

Same direction, newest first

  1. 2023A Long Way to Go: Investigating Length Correlations in RLHF5 Oct
  2. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  3. 2023Reinforced Self-Training (ReST) for Language Modeling17 Aug
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  5. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedback17 May
  6. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  7. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  9. 2022Uncertainty Estimation for Language Reward Models14 Mar
  10. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 20

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  4. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  6. 2020Learning to summarize from human feedbackcited by 2,562
  7. 2016Concrete Problems in AI Safetycited by 1,582
  8. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  9. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  10. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  11. 2019Conformal Prediction Under Covariate Shiftcited by 511
  12. 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 488
8 more
  1. 2020Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awarenesscited by 389
  2. 2021FUDGE: Controlled Text Generation With Future Discriminatorscited by 374
  3. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 321
  4. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  5. 2021Epistemic Neural Networkscited by 120
  6. 2023Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedbackcited by 71
  7. 2020Neural Ensemble Search for Uncertainty Estimation and Dataset Shiftcited by 56
  8. 2022Reward Gaming in Conditional Text Generationcited by 29

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 57; 2025: 71; 2026: 66.

195 papers.

Sort:

By area:

  1. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  2. 2025SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Modelscited by 209
  3. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  4. 2025Scalable Best-of-N Selection for Large Language Models via Self-Certaintycited by 182
  5. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
  6. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
  7. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 139
  8. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHFcited by 133
  9. 2024BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Samplingcited by 120
  10. 2024Theoretical guarantees on the best-of-n alignment policycited by 116
  11. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 115
  12. 2024Fast Best-of-N Decoding via Speculative Rejectioncited by 112
38 more
  1. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  2. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 91
  3. 2024Human Alignment of Large Language Models through Online Preference Optimisationcited by 90
  4. 2025Reward Shaping to Mitigate Reward Hacking in RLHFcited by 88
  5. 2024On Designing Effective RL Reward at Training Time for LLM Reasoningcited by 87
  6. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  7. 2024InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modelingcited by 76
  8. 2024RRM: Robust Reward Model Training Mitigates Reward Hackingcited by 75
  9. 2024Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknowncited by 71
  10. 2024Robust Preference Optimization through Reward Model Distillationcited by 67
  11. 2025Sampling-Efficient Test-Time Scaling: Self-Estimating the Best-of-N Sampling in Early Decodingcited by 62
  12. 2024RMB: Comprehensively Benchmarking Reward Models in LLM Alignmentcited by 60
  13. 2025Multi-Agent Verification: Scaling Test-Time Compute with Multiple Verifierscited by 58
  14. 2025All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuningcited by 58
  15. 2023Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensemblescited by 55
  16. 2024The Importance of Online Data: Understanding Preference Fine-tuning via Coveragecited by 55
  17. 2024Value Augmented Sampling for Language Model Alignment and Personalizationcited by 49
  18. 2024Offline Regularised Reinforcement Learning for Large Language Models Alignmentcited by 48
  19. 2025GRPO-Guard: Mitigating Implicit Over-Optimization in Flow Matching via Regulated Clippingcited by 48
  20. 2024Asymptotics of Language Model Alignmentcited by 46
  21. 2024Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignmentcited by 46
  22. 2024Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemblecited by 43
  23. 2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 43
  24. 2024The Perfect Blend: Redefining RLHF with Mixture of Judgescited by 43
  25. 2024Conditional Language Policy: A General Framework for Steerable Multi-Objective Finetuningcited by 42
  26. 2025Beyond Reward Hacking: Causal Rewards for Large Language Model Alignmentcited by 42
  27. 2025RL Tango: Reinforcing Generator and Verifier Together for Language Reasoningcited by 39
  28. 2024Bayesian Reward Models for LLM Alignmentcited by 35
  29. 2024CREAM: Consistency Regularized Self-Rewarding Language Modelscited by 35
  30. 2024InfAlign: Inference-aware language model alignmentcited by 34
  31. 2024Overcoming Reward Overoptimization via Adversarial Policy Optimization with Lightweight Uncertainty Estimationcited by 33
  32. 2024Reward-Robust RLHF in LLMscited by 32
  33. 2025Inference-Time Reward Hacking in Large Language Modelscited by 32
  34. 2024Post-hoc Reward Calibration: A Case Study on Length Biascited by 29
  35. 2025Beyond Correctness: Harmonizing Process and Outcome Rewards through RL Trainingcited by 29
  36. 2025RewardAnything: Generalizable Principle-Following Reward Modelscited by 27
  37. 2024On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimizationcited by 26
  38. 2025reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputscited by 25

50 most cited of 195 All 195 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Reward Models are Metrics in a Trench Coat
  2. 2024Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignment
  3. 2025Reusing Embeddings: Reproducible Reward Model Research in Large Language Model Alignment without GPUs
  4. 2025Reviving The Classics: Active Reward Modeling in Large Language Model Alignment
  5. 2025Generalist Reward Models: Found Inside Large Language Models
  6. 2026Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewards
  7. 2026Temper and Tilt Lead to SLOP: Reward Hacking Mitigation with Inference-Time Alignment
  8. 2025GRAM: A Generative Foundation Reward Model for Reward Generalization