mlsanityData to 7 Oct 2026

Reward Shaping to Mitigate Reward Hacking in RLHF

arXiv 2502.18770 ↗First posted 26 Feb 2025cs.LG cs.AI cs.CLcited by 88

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement learning from human feedback (RLHF) is widely used to align large language models (LLMs) with human preferences. However, RLHF remains vulnerable to reward hacking, whereby a policy exploits imperfections in the reward function instead of learning the intended behavior, thereby undermining alignment. Although reward shaping can stabilize RLHF training and partially mitigate reward hacking, shaping methods and their underlying design principles have not been systematically investigated. To address this gap, we conduct a comprehensive study of prevalent reward-shaping techniques. Our analysis identifies two key design principles: (1) the reinforcement-learning reward should be bounded, and (2) it should grow rapidly at first and then gradually saturate. Motivated by these principles, we propose Preference as Reward (PAR), a novel method that uses the latent preferences encoded in the reward model as the reinforcement-learning signal. We further show that PAR possesses two variance-reduction properties that stabilize RLHF training and substantially widen the practical window for early stopping. Our evaluation consists of two parts. First, we compare PAR with several other reward-shaping strategies using Proximal Policy Optimization (PPO) as the reinforcement-learning algorithm and Gemma2-2B as the base model. Second, we compare PAR with the vanilla baseline (i.e., unshaped reward) across four base models and four reinforcement-learning algorithms. In the first set of experiments, PAR consistently outperforms other reward-shaping methods and also reflects high data efficiency and robustness. The second set of experiments shows that PAR is particularly effective for actor-critic RL algorithms when value estimates become unstable and demonstrates its effectiveness across different base models. The code is available at https://github.com/PorUna-byte/PAR.

Key facts. Cited by 88 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 40 of its 74 references that are in the map. First posted on arXiv 26 Feb 2025.

Builds on

40 of 74 references.

Closest ancestors · 22

Same direction, newest first

  1. 2025The Energy Loss Phenomenon in RLHF: A New Perspective on Mitigating Reward Hacking31 Jan
  2. 2024Post-hoc Reward Calibration: A Case Study on Length Bias25 Sep
  3. 2024RRM: Robust Reward Model Training Mitigates Reward Hacking20 Sep
  4. 2024Policy Filtration for RLHF to Mitigate Noise in Reward Models11 Sep
  5. 2024Catastrophic Goodhart: regularizing RLHF with KL divergence does not mitigate heavy-tailed reward misspecification19 Jul
  6. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs14 Jun
  7. 2024Scalable Ensembling For Mitigating Reward Overoptimisation3 Jun
  8. 2024Improving Reinforcement Learning from Human Feedback Using Contrastive Rewards12 Mar
  9. 2024InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modeling14 Feb
  10. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHF11 Feb
  11. 2024Transforming and Combining Rewards for Aligning Large Language Models1 Feb
  12. 2024Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemble30 Jan
10 more
  1. 2024WARM: On the Benefits of Weight Averaged Reward Models22 Jan
  2. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking14 Dec
  3. 2023Confronting Reward Model Overoptimization with Constrained RLHF6 Oct
  4. 2023A Long Way to Go: Investigating Length Correlations in RLHF5 Oct
  5. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  6. 2023Secrets of RLHF in Large Language Models Part I: PPO11 Jul
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  8. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  10. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun

Other references · 18

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2015Trust Region Policy Optimizationcited by 2,503
  6. 2016Concrete Problems in AI Safetycited by 1,582
  7. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  8. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  9. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  10. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  11. 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 188
  12. 2024Language Models Learn to Mislead Humans via RLHFcited by 118
6 more
  1. 2024Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinementcited by 111
  2. 2023LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scorescited by 88
  3. 2014Domain-Adversarial Neural Networkscited by 78
  4. 2018Reinforcement Learning for Improving Agent Designcited by 55
  5. 2016Self-Modification of Policy and Utility Function in Rational Agentscited by 11
  6. 2022Robust Losses for Learning Value Functionscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 38; 2026: 50.

88 papers.

Sort:

By area:

  1. 2025SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Modelscited by 209
  2. 2025Self-Rewarding Vision-Language Model via Reasoning Decompositioncited by 95
  3. 2025Self-Adapting Language Modelscited by 71
  4. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 34
  5. 2025Inference-Time Reward Hacking in Large Language Modelscited by 32
  6. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 30
  7. 2025Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learningcited by 28
  8. 2025RewardAnything: Generalizable Principle-Following Reward Modelscited by 27
  9. 2025A Technical Survey of Reinforcement Learning Techniques for Large Language Modelscited by 19
  10. 2025Adaptation of Agentic AI: A Survey of Post-Training, Memory, and Skillscited by 19
  11. 2025Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoningcited by 18
  12. 2026Reward Hacking in Rubric-Based Reinforcement Learningcited by 18
38 more
  1. 2025Virtual Agent Economiescited by 17
  2. 2025A Survey on Agentic Security: Applications, Threats and Defensescited by 17
  3. 2025Encouraging Good Processes Without the Need for Good Answers: Reinforcement Learning for LLM Agent Planningcited by 15
  4. 2025GeoVLM-R1: Reinforcement Fine-Tuning for Improved Remote Sensing Reasoningcited by 15
  5. 2025Environment Scaling for Interactive Agentic Experience Collection: A Surveycited by 13
  6. 2025Doubly Robust Alignment for Large Language Modelscited by 11
  7. 2025AutoRule: Reasoning Chain-of-thought Extracted Rule-based Rewards Improve Preference Learningcited by 11
  8. 2025Pluralistic Off-policy Evaluation and Alignmentcited by 11
  9. 2025Bradley-Terry and Multi-Objective Reward Modeling Are Complementarycited by 10
  10. 2025Truthful or Fabricated? Using Causal Attribution to Mitigate Reward Hacking in Explanationscited by 9
  11. 2026Reproducing, Analyzing, and Detecting Reward Hacking in Rubric-Based Reinforcement Learningcited by 9
  12. 2025Enhancing Tool Learning in Large Language Models with Hierarchical Error Checklistscited by 7
  13. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 7
  14. 2026Tackling Length Inflation Without Trade-offs: Group Relative Reward Rescaling for Reinforcement Learningcited by 7
  15. 2026EvoRubric: Self-Evolving Rubric-Driven RL for Open-Ended Generationcited by 7
  16. 2026How to Train Your Deep Research Agent? Prompt, Reward, and Policy Optimization in Search-R1cited by 6
  17. 2025Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignmentcited by 5
  18. 2025Alleviating Attention Hacking in Discriminative Reward Modeling through Interaction Distillationcited by 5
  19. 2025ARM-FM: Automated Reward Machines via Foundation Models for Compositional Reinforcement Learningcited by 5
  20. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 5
  21. 2026Diagnosing Training Inference Mismatch in LLM Reinforcement Learning via a Zero-Mismatch Referencecited by 5
  22. 2025Toward Evaluative Thinking: Meta Policy Optimization with Evolving Reward Modelscited by 4
  23. 2025Position: Biology is the Challenge Physics-Informed ML Needs to Evolvecited by 4
  24. 2026Detecting and Suppressing Reward Hacking with Gradient Fingerprintscited by 4
  25. 2026Uncertainty-Aware Reward Modeling for Stable RLHFcited by 4
  26. 2025Generative Adversarial Post-Training Mitigates Reward Hacking in Live Human-AI Music Interactioncited by 3
  27. 2026SAIL: Self-Amplified Iterative Learning for Diffusion Model Alignment with Minimal Human Feedbackcited by 3
  28. 2026One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Modelscited by 3
  29. 2026Explaining and Preventing Alignment Collapse in Iterative RLHFcited by 3
  30. 2026Optimal Transport for LLM Reward Modeling from Noisy Preferencecited by 3
  31. 2026The Dark Room in the Reward Channel: Dense Prediction Rewards Collapse GRPO-Trained LLM Agents -- and The Channel, Not the Content, Decides What Workscited by 3
  32. 2025Learning Explainable Dense Reward Shapes via Bayesian Optimizationcited by 2
  33. 2026Reward Shaping for (Inference-Time) Alignment: A Stackelberg Game Perspectivecited by 2
  34. 2026ClinAlign: Scaling Healthcare Alignment from Clinician Preferencecited by 2
  35. 2026Engineering Reasoning and Instruction (ERI) Benchmark: A Large Taxonomy-driven Dataset for Foundation Models and Agentscited by 2
  36. 2026Towards Robust Endogenous Reasoning: Unifying Drift Adaptation in Non-Stationary Tuningcited by 2
  37. 2026Towards Robust LLM Post-Training: Automatic Failure Management for Reinforcement Fine-Tuningcited by 2
  38. 2026Multimodal Reward Hacking in Reinforcement Learningcited by 2

50 most cited of 88 All 88 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling
  2. 2026Meta-Learned Reward Shaping for Reinforcement Learning from Human Feedback
  3. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  4. 2024Optimal Design for Reward Modeling in RLHF
  5. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  6. 2025A Unified Pairwise Framework for RLHF: Bridging Generative Reward Modeling and Policy Optimization
  7. 2026Generalisation of RLHF under Reward Shift and Clipped KL Regularisation
  8. 2025Influencing Humans to Conform to Preference Models for RLHF