mlsanityData to 14 Oct 2026

RMB: Reward Model Boosting Mitigates Reward Hacking

arXiv 2609.33221 ↗First posted 27 Sep 2026cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) is a powerful technique for aligning large language models (LLMs) with human preference. However, it often suffers from the reward hacking issue, where policy optimization improves the proxy reward model while actually degrading performance with respect to the true human preference, due to the imperfection of the proxy. To address this, we propose Reward Model Boosting (RMB), a novel approach that enhances the robustness and reliability of the reward signal for RLHF. RMB first trains a set of reward models with a diversity-promoting regularizer. This encourages each model to learn complementary aspects of the reward landscape. Then, RMB learns a lightweight aggregator in the principle of boosting to aggregate the outputs of the diverse reward models into a more accurate and robust reward signal. Our extensive experiments demonstrate that RMB significantly improves reward accuracy on both in-distribution and out-of-distribution datasets, substantially mitigating the reward hacking issue and ultimately improving RLHF performance.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 55 of its 86 references that are in the map. First posted on arXiv 27 Sep 2026.

Builds on

55 of 86 references.

Closest ancestors · 24

Same direction, newest first

  1. 2025Bradley-Terry and Multi-Objective Reward Modeling Are Complementary10 Jul
  2. 2025Reward Model Overoptimisation in Iterated RLHF23 May
  3. 2025Mitigating Reward Over-Optimization in RLHF via Behavior-Supported Regularization23 Mar
  4. 2025reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputs14 Mar
  5. 2025Distributionally Robust Reinforcement Learning with Human Feedback1 Mar
  6. 2024RRM: Robust Reward Model Training Mitigates Reward Hacking20 Sep
  7. 2024Reward-Robust RLHF in LLMs18 Sep
  8. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts18 Jun
  9. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs14 Jun
  10. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer26 May
  11. 2024RLHF Workflow: From Reward Modeling to Online RLHF13 May
  12. 2024RewardBench: Evaluating Reward Models for Language Modeling20 Mar
12 more
  1. 2024Overcoming Reward Overoptimization via Adversarial Policy Optimization with Lightweight Uncertainty Estimation8 Mar
  2. 2024InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modeling14 Feb
  3. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHF11 Feb
  4. 2024Corruption Robust Offline Reinforcement Learning with Human Feedback9 Feb
  5. 2024WARM: On the Benefits of Weight Averaged Reward Models22 Jan
  6. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling11 Jan
  7. 2023Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensembles30 Dec
  8. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking14 Dec
  9. 2023Confronting Reward Model Overoptimization with Constrained RLHF6 Oct
  10. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  11. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  12. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr

Other references · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2017Proximal Policy Optimization Algorithmscited by 15,215
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
  7. 2024Qwen2.5 Technical Reportcited by 4,942
  8. 2020Learning to summarize from human feedbackcited by 2,588
  9. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  10. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  11. 2018On Lazy Training in Differentiable Programmingcited by 675
  12. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 671
19 more
  1. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 486
  2. 2017Synthetic and Natural Noise Both Break Neural Machine Translationcited by 306
  3. 2023Mitigating the Alignment Tax of RLHFcited by 183
  4. 2019Combating Adversarial Misspellings with Robust Word Recognitioncited by 163
  5. 2016Unbiased Learning-to-Rank with Biased Feedbackcited by 157
  6. 2023Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarizationcited by 70
  7. 2024Correlated Proxies: A New Definition and Improved Mitigation for Reward Hackingcited by 50
  8. 2023Corruption-Robust Offline Reinforcement Learning with General Function Approximationcited by 31
  9. 2018Applying Deep Learning To Airbnb Searchcited by 24
  10. 2025Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Modelscited by 23
  11. 2025Adversarial Training of Reward Modelscited by 18
  12. 2021Are Neural Ranking Models Robust?cited by 16
  13. 2026Reward-free Alignment for Conflicting Objectivescited by 11
  14. 2024Towards Continual Learning Desiderata via HSIC-Bottleneck Orthogonalization and Equiangular Embeddingcited by 10
  15. 2022Learning To Rank Diversely At Airbnbcited by 6
  16. 2023Diversity-enhancing Generative Network for Few-shot Hypothesis Adaptationcited by 4
  17. 2022Which Tricks Are Important for Learning to Rank?cited by 3
  18. 2025KETCHUP: K-Step Return Estimation for Sequential Knowledge Distillationcited by 3
  19. 2021Prioritizing Original News on Facebookcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  2. 2025A Unified Pairwise Framework for RLHF: Bridging Generative Reward Modeling and Policy Optimization
  3. 2024Prototypical Reward Network for Data-Efficient RLHF
  4. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style
  5. 2024How to Evaluate Reward Models for RLHF
  6. 2026Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewards
  7. 2024Optimal Design for Reward Modeling in RLHF
  8. 2024Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback