mlsanityData to 14 Oct 2026

Correlated Proxies: A New Definition and Improved Mitigation for Reward Hacking

arXiv 2403.03185 ↗First posted 5 Mar 2024cs.LG cs.AIcited by 50

On the map: LLM alignment / RLHF / preference optimisation → Reward hacking and misspecification

Abstract (arXiv)

Because it is difficult to precisely specify complex objectives, reinforcement learning policies are often optimized using proxy reward functions that only approximate the true goal. However, optimizing proxy rewards frequently leads to reward hacking: the optimized reward function ceases to be a good proxy and the resulting policy performs poorly with respect to the unspecified true reward. Principled solutions to reward hacking have been impeded by the lack of a good definition for the problem. To address this gap, we introduce a definition of reward hacking based on the correlation between proxy and true rewards for states and actions seen by a "reference policy" that breaks down under optimization. We show that this definition captures reward hacking behavior across several realistic settings, including in reinforcement learning from human feedback (RLHF). Using our formulation, we show theoretically that regularization to the reference policy can effectively prevent reward hacking. While the current practice in RLHF applies a KL penalty between action distributions for this purpose, our theory suggests regularizing the chi2 divergence between the policies' occupancy measures can be more effective. We intuitively show the benefits of this type of regularization and demonstrate that it better mitigates reward hacking in practice across four realistic settings, including RLHF. Our code is available at https://github.com/cassidylaidlaw/orpo.

Key facts. Cited by 50 papers in the map (data through 14 Oct 2026). Placed in Reward hacking and misspecification, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 64 of its 92 references that are in the map. First posted on arXiv 5 Mar 2024.

Builds on

64 of 92 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Defining and Characterizing Reward Hacking27 Sep
  2. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models10 Jan

Other references · 62

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  6. 2020Learning to summarize from human feedbackcited by 2,588
  7. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  8. 2016Concrete Problems in AI Safetycited by 1,642
  9. 2016Generative Adversarial Imitation Learningcited by 1,597
  10. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  11. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  12. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
38 more
  1. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  2. 2018Scalable agent alignment via reward modeling: a research directioncited by 447
  3. 2017Algorithmic decision making and the cost of fairnesscited by 403
  4. 2018Reward learning from human preferences and demonstrations in Ataricited by 316
  5. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 302
  6. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 297
  7. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 279
  8. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 269
  9. 2022The Alignment Problem from a Deep Learning Perspectivecited by 258
  10. 2017Inverse Reward Designcited by 249
  11. 2018Safe Exploration in Continuous Action Spacescited by 247
  12. 2018Provably Efficient Maximum Entropy Explorationcited by 242
  13. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 231
  14. 2019Efficient Exploration via State Marginal Matchingcited by 188
  15. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraintscited by 174
  16. 2018Categorizing Variants of Goodhart's Lawcited by 149
  17. 2019Lyapunov-based Safe Policy Optimization for Continuous Controlcited by 145
  18. 2022RL with KL penalties is better viewed as Bayesian inferencecited by 135
  19. 2020Reward-rational (implicit) choice: A unifying formalism for reward learningcited by 120
  20. 2023Aligning Language Models with Preferences through f-divergence Minimizationcited by 120
  21. 2019Optimal Policies Tend to Seek Powercited by 119
  22. 2022Jump-Start Reinforcement Learningcited by 112
  23. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learningcited by 111
  24. 2021Reward (Mis)design for Autonomous Drivingcited by 104
  25. 2023LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Modelscited by 88
  26. 2017Flow: A Modular Learning Framework for Mixed Autonomy Trafficcited by 73
  27. 2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimationcited by 71
  28. 2020Quantifying Differences in Reward Functionscited by 63
  29. 2022Causal Confusion and Reward Misidentification in Preference-Based Reward Learningcited by 53
  30. 2021Twice regularized MDPs and the equivalence between robustness and regularizationcited by 47
  31. 2019Learning Human Objectives by Evaluating Hypothetical Behaviorcited by 41
  32. 2018Penalizing side effects using stepwise relative reachabilitycited by 38
  33. 2020Leverage the Average: an Analysis of KL Regularization in RLcited by 33
  34. 2021Direct Behavior Specification via Constrained Reinforcement Learningcited by 31
  35. 2023Bridging RL Theory and Practice with the Effective Horizoncited by 31
  36. 2022The Challenge of Understanding What Users Want: Inconsistent Preferences and Engagement Optimizationcited by 29
  37. 2022Performative Reinforcement Learningcited by 27
  38. 2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferencescited by 27

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 17; 2026: 30.

50 papers.

Sort:

By area:

  1. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 384
  2. 2025Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoningcited by 92
  3. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  4. 2025Inference-Time Reward Hacking in Large Language Modelscited by 34
  5. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 33
  6. 2025CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMscited by 31
  7. 2026SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agentscited by 28
  8. 2025GARDO: Reinforcing Diffusion Models without Reward Hackingcited by 17
  9. 2026Adaptive Rollout Allocation for Online Reinforcement Learning with Verifiable Rewardscited by 17
  10. 2026Countdown-Code: A Testbed for Studying The Emergence and Generalization of Reward Hacking in RLVRcited by 17
  11. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 15
  12. 2025Recontextualization Mitigates Specification Gaming without Modifying the Specificationcited by 14
38 more
  1. 2025Flattery, Fluff, and Fog: Diagnosing and Mitigating Idiosyncratic Biases in Preference Modelscited by 13
  2. 2025Doubly Robust Alignment for Large Language Modelscited by 12
  3. 2025Bradley-Terry and Multi-Objective Reward Modeling Are Complementarycited by 11
  4. 2024Hummer: Towards Limited Competitive Preference Datasetcited by 10
  5. 2025Towards a Sharp Analysis of Offline Policy Learning for f-Divergence-Regularized Contextual Banditscited by 10
  6. 2025VRPO: Rethinking Value Modeling for Robust RL under Noisy Supervision in LLM Post-Trainingcited by 9
  7. 2024The Perils of Optimizing Learned Reward Functions: Low Training Error Does Not Guarantee Low Regretcited by 6
  8. 2026Efficient RLVR Training via Weighted Mutual Information Data Selectioncited by 6
  9. 2025Shallow Preference Signals: Large Language Model Aligns Even Better with Truncated Data?cited by 5
  10. 2026Improved techniques for fine-tuning flow models via adjoint matching: a deterministic control pipelinecited by 5
  11. 2026Imperfect World Models are Exploitablecited by 4
  12. 2026Reward Hacking in Language Model Agents: Revisiting AI Safety Gridworldscited by 4
  13. 2026Debate Training Reduces Reward Hacking in RLAIFcited by 4
  14. 2024CANDERE-COACH: Reinforcement Learning from Noisy Feedbackcited by 3
  15. 2026Best-of-Tails: Bridging Optimism and Pessimism in Inference-Time Alignmentcited by 3
  16. 2026From Curiosity to Caution: Mitigating Reward Hacking for Best-of-N with Pessimismcited by 3
  17. 2026CoAct: Co-Active LLM Preference Learning with Human-AI Synergycited by 3
  18. 2026Engineering Reliable Coding Agents: Evaluating and Operating the System Around the Modelcited by 3
  19. 2025Review of Hallucination Understanding in Large Language and Vision Modelscited by 2
  20. 2026Theoretical Foundations and Effective Algorithms for Policy-Aware Simulator Learningcited by 2
  21. 2025Residual Reward Models: Leveraging Prior Knowledge for Efficient Preference-based Reinforcement Learning in Roboticscited by 1
  22. 2025Repairing Reward Functions with Feedback to Mitigate Reward Hackingcited by 1
  23. 2026Consequentialist Objectives and Catastrophecited by 1
  24. 2026ConsistRM: Improving Generative Reward Models via Consistency-Aware Self-Trainingcited by 1
  25. 2026Do Prompt-Elicited Trajectories Reflect Training-Time Reward Hacking? A Systematic Study on Monitoring Training-Time Reward Hacking in Code Generationcited by 1
  26. 2026Against Proxy Optimizationcited by 1
  27. 2026Norm Enforcement for AI Agents: Robustly Shaping Behavior in Multi-Agent Systemscited by 1
  28. 2026Explore or Converge? Stage-Guided Per-Step Optimization for Diffusion Modelscited by 1
  29. 2026Pedagogical Safety in Educational Reinforcement Learning: Formalizing and Detecting Reward Hacking in AI Tutoring Systemscited by 0
  30. 2026Directional Alignment Mitigates Reward Hacking in Reinforcement Learning for Language Modelscited by 0
  31. 2026When are LLMs Sufficient Policy Optimizers for Sequential RL Tasks?cited by 0
  32. 2026Modification-Considering Value Learning for Reward Hacking Mitigation in RLcited by 0
  33. 2026TACS: Trajectory-Aware Candidate Selection for LLM Jailbreak Suffix Optimizationcited by 0
  34. 2026When Better Gets Worse: Improvement Fidelity for Self-Improving Agents in Adaptive Worldscited by 0
  35. 2026RMB: Reward Model Boosting Mitigates Reward Hackingcited by 0
  36. 2026Verifier Errors in RLVR: Reward Hacking, Limits of Feedback, and Selective Controlcited by 0
  37. 2026Revisiting scaling laws for reward optimizationcited by 0
  38. 2026CATCH: A Controllable Analysis Testbed for Reward Hacking in Coding RLcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Robust Optimization for Mitigating Reward Hacking with Correlated Proxies
  2. 2026Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewards
  3. 2025Going Beyond Heuristics by Imposing Policy Improvement as a Constraint
  4. 2025Information-Theoretic Reward Modeling for Stable RLHF: Detecting and Mitigating Reward Hacking
  5. 2026From Rebound to Remedy: Understanding and Mitigating Reward Hacking via Representation Engineering
  6. 2026Reward Hacking in Rubric-Based Reinforcement Learning
  7. 2025Learning a Pessimistic Reward Model in RLHF
  8. 2025The Energy Loss Phenomenon in RLHF: A New Perspective on Mitigating Reward Hacking