mlsanityData to 7 Oct 2026

Defining and Characterizing Reward Hacking

arXiv 2209.13085 ↗First posted 27 Sep 2022cs.LG stat.MLcited by 184

On the map: LLM alignment / RLHF / preference optimisation → Reward hacking and misspecification

Abstract (arXiv)

We provide the first formal definition of reward hacking, a phenomenon where optimizing an imperfect proxy reward function leads to poor performance according to the true reward function. We say that a proxy is unhackable if increasing the expected proxy return can never decrease the expected true return. Intuitively, it might be possible to create an unhackable proxy by leaving some terms out of the reward function (making it "narrower") or overlooking fine-grained distinctions between roughly equivalent outcomes, but we show this is usually not the case. A key insight is that the linearity of reward (in state-action visit counts) makes unhackability a very strong condition. In particular, for the set of all stochastic policies, two reward functions can only be unhackable if one of them is constant. We thus turn our attention to deterministic policies and finite sets of stochastic policies, where non-trivial unhackable pairs always exist, and establish necessary and sufficient conditions for the existence of simplifications, an important special case of unhackability. Our results reveal a tension between using reward functions to specify narrow tasks and aligning AI systems with human values.

Key facts. Cited by 184 papers in the map (data through 7 Oct 2026). Placed in Reward hacking and misspecification, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 24 of its 59 references that are in the map. First posted on arXiv 27 Sep 2022.

Builds on

24 of 59 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models10 Jan

Other references · 23

  1. 2020Learning to summarize from human feedbackcited by 2,562
  2. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  3. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  4. 2020Artificial Intelligence, Values and Alignmentcited by 443
  5. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  6. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
  7. 2016Cooperative Inverse Reinforcement Learningcited by 298
  8. 2017Inverse Reward Designcited by 246
  9. 2018Categorizing Variants of Goodhart's Lawcited by 132
  10. 2019Optimal Policies Tend to Seek Powercited by 115
  11. 2019Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspectivecited by 107
  12. 2016The Off-Switch Gamecited by 97
11 more
  1. 2021Consequences of Misaligned AIcited by 79
  2. 2021On the Expressivity of Markov Rewardcited by 71
  3. 2017Reinforcement Learning with a Corrupted Reward Channelcited by 68
  4. 2021On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learningcited by 59
  5. 2018Penalizing side effects using stepwise relative reachabilitycited by 38
  6. 2020The Problem with Metrics is a Fundamental Problem for AIcited by 37
  7. 2019Conservative Agency via Attainable Utility Preservationcited by 35
  8. 2021Hard Choices in Artificial Intelligencecited by 26
  9. 2020Value Alignment Verificationcited by 20
  10. 2020Avoiding Tampering Incentives in Deep RL via Decoupled Approvalcited by 12
  11. 2020REALab: An Embedded Perspective on Tamperingcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 18; 2024: 14; 2025: 41; 2026: 108.

184 papers.

Sort:

By area:

  1. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  2. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 767
  3. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  4. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  5. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  6. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
  7. 2024LLM Critics Help Catch LLM Bugscited by 138
  8. 2023Harms from Increasingly Agentic Algorithmic Systemscited by 134
  9. 2023Motif: Intrinsic Motivation from Artificial Intelligence Feedbackcited by 99
  10. 2023Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedbackcited by 78
  11. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  12. 2024Correlated Proxies: A New Definition and Improved Mitigation for Reward Hackingcited by 50
38 more
  1. 2023Learning to Generate Better Than Your LLMcited by 49
  2. 2023Zero-Shot Goal-Directed Dialogue via RL on Imagined Conversationscited by 38
  3. 2025Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Modelscited by 35
  4. 2024Best-of-Venom: Attacking RLHF by Injecting Poisoned Preference Datacited by 33
  5. 2024AI Safety in Generative AI Large Language Models: A Surveycited by 31
  6. 2025ReSo: A Reward-driven Self-organizing LLM-based Multi-Agent System for Reasoning Taskscited by 30
  7. 2022Reward Gaming in Conditional Text Generationcited by 29
  8. 2024Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree?cited by 29
  9. 2025A Survey on Autonomy-Induced Security Risks in Large Model-Based Agentscited by 28
  10. 2022Misspecification in Inverse Reinforcement Learningcited by 27
  11. 2024FuRL: Visual-Language Models as Fuzzy Rewards for Reinforcement Learningcited by 26
  12. 2023Improving Generalization of Alignment with Human Preferences through Group Invariant Learningcited by 25
  13. 2025Is It Thinking or Cheating? Detecting Implicit Reward Hacking by Measuring Reasoning Effortcited by 25
  14. 2025CodeV: Code with Images for Faithful Visual Reasoning via Tool-Aware Policy Optimizationcited by 24
  15. 2026LLMs Gaming Verifiers: RLVR can Lead to Reward Hackingcited by 24
  16. 2023On The Fragility of Learned Reward Functionscited by 21
  17. 2023Searching for High-Value Molecules Using Reinforcement Learning and Transformerscited by 18
  18. 2025Stepwise Guided Policy Optimization: Coloring your Incorrect Reasoning in GRPOcited by 18
  19. 2026Intelligent AI Delegationcited by 16
  20. 2023STARC: A General Framework For Quantifying Differences Between Reward Functionscited by 15
  21. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Designcited by 15
  22. 2025Dark Patterns Meet GUI Agents: LLM Agent Susceptibility to Manipulative Interfaces and the Role of Human Oversightcited by 15
  23. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 14
  24. 2025Learn to Guide Your Diffusion Modelcited by 12
  25. 2024On the Limitations of Markovian Rewards to Express Multi-Objective, Risk-Sensitive, and Modal Taskscited by 9
  26. 2026LLMs Corrupt Your Documents When You Delegatecited by 9
  27. 2026A Systematic Survey of Security Threats and Defenses in LLM-Based AI Agents: A Layered Attack Surface Frameworkcited by 9
  28. 2025Societal Alignment Frameworks Can Improve LLM Alignmentcited by 8
  29. 2025VARP: Reinforcement Learning from Vision-Language Model Feedback with Agent Regularized Preferencescited by 8
  30. 2023Active teacher selection for reward learningcited by 7
  31. 2024BRAIn: Bayesian Reward-conditioned Amortized Inference for natural language generation from feedbackcited by 7
  32. 2025MR. Judge: Multimodal Reasoner as a Judgecited by 7
  33. 2025PentestJudge: Judging Agent Behavior Against Operational Requirementscited by 7
  34. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 7
  35. 2025Understanding Sampler Stochasticity in Training Diffusion Models for RLHFcited by 7
  36. 2024DACO: Towards Application-Driven and Comprehensive Data Analysis via Code Generationcited by 6
  37. 2025Specification Self-Correction: Mitigating In-Context Reward Hacking Through Test-Time Refinementcited by 6
  38. 2026The Obfuscation Atlas: Mapping Where Honesty Emerges in RLVR with Deception Probescited by 6

50 most cited of 184 All 184 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Reward Hacking as Equilibrium under Finite Evaluation
  2. 2026Robust Optimization for Mitigating Reward Hacking with Correlated Proxies
  3. 2026Reward Hacking in Language Model Agents: Revisiting AI Safety Gridworlds
  4. 2020Pitfalls of learning a reward function online
  5. 2026Countdown-Code: A Testbed for Studying The Emergence and Generalization of Reward Hacking in RLVR
  6. 2026Proxy Reward Internalization and Mechanistic Exploitation: A Learned Precursor to Reward Hacking and Its Generalization
  7. 2026Optimizing the Score, Losing Sight of the Task: Reward Hacking Across Weights, Selection, and Prompts
  8. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challenges