Data to 14 Oct 2026

RRM: Robust Reward Model Training Mitigates Reward Hacking

arXiv 2409.13156 ↗First posted 20 Sep 2024cs.CLcited by 76

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reward models (RMs) play a pivotal role in aligning large language models (LLMs) with human preferences. However, traditional RM training, which relies on response pairs tied to specific prompts, struggles to disentangle prompt-driven preferences from prompt-independent artifacts, such as response length and format. In this work, we expose a fundamental limitation of current RM training methods, where RMs fail to effectively distinguish between contextual signals and irrelevant artifacts when determining preferences. To address this, we introduce a causal framework that learns preferences independent of these artifacts and propose a novel data augmentation technique designed to eliminate them. Extensive experiments show that our approach successfully filters out undesirable artifacts, yielding a more robust reward model (RRM). Our RRM improves the performance of a pairwise reward model trained on Gemma-2-9b-it, on RewardBench, increasing accuracy from 80.61% to 84.15%. Additionally, we train two DPO policies using both the RM and RRM, demonstrating that the RRM significantly enhances DPO-aligned policies, improving MT-Bench scores from 7.27 to 8.31 and length-controlled win-rates in AlpacaEval-2 from 33.46% to 52.49%.

Key facts. Cited by 76 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 48 of its 64 references that are in the map. First posted on arXiv 20 Sep 2024.

Builds on

48 of 64 references.

Closest ancestors · 22

Same direction, newest first

10 more

Other references · 26

2023GPT-4 Technical Reportcited by 18,396
2022Training language models to follow instructions with human feedbackcited by 15,971
2017Proximal Policy Optimization Algorithmscited by 15,215
2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
2020Learning to summarize from human feedbackcited by 2,588
2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748

14 more

2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
2023Statistical Rejection Sampling Improves Preference Optimizationcited by 323
2021Understanding Dataset Difficulty with V-Usable Informationcited by 291
2024Direct Language Model Alignment from Online AI Feedbackcited by 260
2024Disentangling Length from Quality in Direct Preference Optimizationcited by 196
2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 186
2024Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Modelscited by 178
2024BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Samplingcited by 121
2024LiPO: Listwise Preference Optimization through Learning-to-Rankcited by 96
2024Human Alignment of Large Language Models through Online Preference Optimisationcited by 91
2019On the Weaknesses of Reinforcement Learning for Neural Machine Translationcited by 82
2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66
2024Offline Regularised Reinforcement Learning for Large Language Models Alignmentcited by 48
2023Predicting Text Preference Via Structured Comparative Reasoningcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 38; 2026: 35.

76 papers.

Sort:

By area:

2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 135
2025Reward Shaping to Mitigate Reward Hacking in RLHFcited by 88
2024Self-Generated Critiques Boost Reward Modeling for Language Modelscited by 71
2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
2025AdvancedIF: Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Followingcited by 51
2025Inference-Time Reward Hacking in Large Language Modelscited by 34
2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 33
2025Diversity-Incentivized Exploration for Versatile Reasoningcited by 28
2024CARMO: Dynamic Criteria Generation for Context-Aware Reward Modellingcited by 22
2025Virtual Agent Economiescited by 21
2024RainbowPO: A Unified Framework for Combining Improvements in Preference Optimizationcited by 19
2025Robust Reward Modeling via Causal Rubricscited by 19

38 more

2025Adversarial Training of Reward Modelscited by 18
2025Encouraging Good Processes Without the Need for Good Answers: Reinforcement Learning for LLM Agent Planningcited by 17
2025Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Modelscited by 16
2025Pre-Trained Policy Discriminators are General Reward Modelscited by 15
2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 15
2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
2025On the Robustness of Reward Models for Language Model Alignmentcited by 13
2025Flattery, Fluff, and Fog: Diagnosing and Mitigating Idiosyncratic Biases in Preference Modelscited by 13
2025Environment Scaling for Interactive Agentic Experience Collection: A Surveycited by 13
2026Real-Time Aligned Reward Model beyond Semanticscited by 13
2025Bradley-Terry and Multi-Objective Reward Modeling Are Complementarycited by 11
2026Reward Modeling from Natural Language Human Feedbackcited by 11
2026Adversarial Reward Auditing for Active Detection and Mitigation of Reward Hackingcited by 11
2026Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimizationcited by 9
2025Bias Fitting to Mitigate Length Bias of Reward Model in RLHFcited by 7
2025Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunitiescited by 7
2026Tackling Length Inflation Without Trade-offs: Group Relative Reward Rescaling for Reinforcement Learningcited by 7
2025Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignmentcited by 6
2025Preference Learning for AI Alignment: a Causal Perspectivecited by 6
2025Activation Reward Models for Few-Shot Model Alignmentcited by 6
2025APLOT: Robust Reward Modeling via Adaptive Preference Learning with Optimal Transportcited by 6
2025Rectifying Shortcut Behaviors in Preference-based Reward Learningcited by 6
2025Mitigating Length Bias in RLHF through a Causal Lenscited by 6
2026Uncertainty-Aware Reward Modeling for Stable RLHFcited by 6
2025Information-Theoretic Reward Decomposition for Generalizable RLHFcited by 5
2025Toward Evaluative Thinking: Meta Policy Optimization with Evolving Reward Modelscited by 5
2025Learning a Pessimistic Reward Model in RLHFcited by 5
2025CoLD: Counterfactually-Guided Length Debiasing for Process Reward Models in Mathematical Reasoningcited by 5
2025Causal Reward Adjustment: Mitigating Reward Hacking in External Reasoning via Backdoor Correctioncited by 5
2025Information-Theoretic Reward Modeling for Stable RLHF: Detecting and Mitigating Reward Hackingcited by 5
2026Spurious Correlation Learning in Preference Optimization: Mechanisms, Consequences, and Mitigation via Tie Trainingcited by 5
2025Fixing the Broken Compass: Diagnosing and Improving Inference-Time Reward Modelingcited by 4
2025Energy-Based Reward Models for Robust Language Model Alignmentcited by 4
2025Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanismcited by 4
2026Automatically Finding Reward Model Biasescited by 4
2026SAIL: Self-Amplified Iterative Learning for Diffusion Model Alignment with Minimal Human Feedbackcited by 3
2026One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Modelscited by 3
2026C2: Scalable Rubric-Augmented Reward Modeling from Binary Preferencescited by 3

50 most cited of 76 All 76 (JSON)

Similar, not cited

Close in wording, no citation either way