mlsanityData to 14 Oct 2026

Goodhart's Law in Reinforcement Learning

arXiv 2310.09144 ↗First posted 13 Oct 2023cs.LGcited by 41

Not on the map. Why?

Abstract (arXiv)

Implementing a reward function that perfectly captures a complex task in the real world is impractical. As a result, it is often appropriate to think of the reward function as a proxy for the true objective rather than as its definition. We study this phenomenon through the lens of Goodhart's law, which predicts that increasing optimisation of an imperfect proxy beyond some critical point decreases performance on the true objective. First, we propose a way to quantify the magnitude of this effect and show empirically that optimising an imperfect proxy reward often leads to the behaviour predicted by Goodhart's law for a wide range of environments and reward functions. We then provide a geometric explanation for why Goodhart's law occurs in Markov decision processes. We use these theoretical insights to propose an optimal early stopping method that provably avoids the aforementioned pitfall and derive theoretical regret bounds for this method. Moreover, we derive a training method that maximises worst-case reward, for the setting where there is uncertainty about the true reward function. Finally, we evaluate our early stopping method experimentally. Our results support a foundation for a theoretically-principled study of reinforcement learning under reward misspecification.

Key facts. Cited by 41 papers in the map (data through 14 Oct 2026). Builds on 17 of its 28 references that are in the map. First posted on arXiv 13 Oct 2023.

Builds on

17 of 28 references.

References · 17

  1. 2017Deep reinforcement learning from human preferencescited by 3,912
  2. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  3. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 365
  4. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
  5. 2018Reward learning from human preferences and demonstrations in Ataricited by 316
  6. 2018Categorizing Variants of Goodhart's Lawcited by 149
  7. 2021Reward (Mis)design for Autonomous Drivingcited by 104
  8. 2019Observational Overfitting in Reinforcement Learningcited by 91
  9. 2021Consequences of Misaligned AIcited by 80
  10. 2017Reinforcement Learning with a Corrupted Reward Channelcited by 70
  11. 2020Quantifying Differences in Reward Functionscited by 63
  12. 2022Invariance in Policy Optimisation and Partial Identifiability in Reward Learningcited by 46
5 more
  1. 2017Occam's razor is insufficient to infer the preferences of irrational agentscited by 45
  2. 2022Reward Gaming in Conditional Text Generationcited by 30
  3. 2022Misspecification in Inverse Reinforcement Learningcited by 27
  4. 2023STARC: A General Framework For Quantifying Differences Between Reward Functionscited by 15
  5. 2020Causal Campbell-Goodhart's law and Reinforcement Learningcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 6; 2026: 27.

41 papers.

Sort:

By area:

  1. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 135
  2. 2024Gradient Routing: Masking Gradients to Localize Computation in Neural Networkscited by 34
  3. 2025Inference-Time Reward Hacking in Large Language Modelscited by 34
  4. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 33
  5. 2024Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree?cited by 31
  6. 2025Think Smarter not Harder: Adaptive Reasoning with Inference Aware Optimizationcited by 29
  7. 2024Automating the Search for Artificial Life with Foundation Modelscited by 20
  8. 2026Countdown-Code: A Testbed for Studying The Emergence and Generalization of Reward Hacking in RLVRcited by 17
  9. 2024Benchmark Inflation: Revealing LLM Performance Gaps Using Retro-Holdoutscited by 14
  10. 2024Can a Bayesian Oracle Prevent Harm from an Agent?cited by 13
  11. 2025Data-regularized Reinforcement Learning for Diffusion Models at Scalecited by 13
  12. 2026The Obfuscation Atlas: Mapping Where Honesty Emerges in RLVR with Deception Probescited by 12
29 more
  1. 2026Understanding Diversity Collapse in RLVR via the Lens of Overtrainingcited by 10
  2. 2026Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalizationcited by 7
  3. 2024The Perils of Optimizing Learned Reward Functions: Low Training Error Does Not Guarantee Low Regretcited by 6
  4. 2025Rethinking Reward Signals in Video GRPO: When Scores Become Targetscited by 6
  5. 2026Test-Driven AI Agent Definition (TDAD): Compiling Tool-Using Agents from Behavioral Specificationscited by 6
  6. 2024ViUniT: Visual Unit Tests for More Robust Visual Programmingcited by 4
  7. 2026Survive or Collapse: The Asymmetric Roles of Data Gating and Reward Grounding in Self-Play RLcited by 4
  8. 2026Likelihood Hacking in Probabilistic Program Synthesiscited by 2
  9. 2026Reward-Aware Trajectory Shaping for Few-step Visual Generationcited by 2
  10. 2026Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluationscited by 2
  11. 2025Think, Act, Learn: A Framework for Autonomous Robotic Agents using Closed-Loop Large Language Modelscited by 1
  12. 2026Oracle-Robust Online Alignment for Large Language Modelscited by 1
  13. 2026Consequentialist Objectives and Catastrophecited by 1
  14. 2026A Sober Look at Agentic Misalignment in Automated Workflowscited by 1
  15. 2026Workflow Closure Is Not Scientific Closure in Auto-Research Systemscited by 1
  16. 2026Against Proxy Optimizationcited by 1
  17. 2024Extinction Risks from AI: Invisible to Science?cited by 0
  18. 2026Why Code, Why Now: An Information-Theoretic Perspective on the Limits of Machine Learningcited by 0
  19. 2026Self++: Co-Determined Agency for Human--AI Symbiosis in Extended Realitycited by 0
  20. 2026Self-Improving Tabular Language Models via Iterative Reward-Guided Post-Trainingcited by 0
  21. 2026Wasserstein Distributionally Robust Regret Optimization for Reinforcement Learning from Human Feedbackcited by 0
  22. 2026Mechanical Enforcement for LLM Governance:Evidence of Governance-Task Decoupling in Financial Decision Systemscited by 0
  23. 2026Reframing AI Loss of Control: What Control Is, How to Have It, How to Lose Itcited by 0
  24. 2026CHASE: How Content Ecosystems Are Reshaped When Ranking Is the Only Targetcited by 0
  25. 2026When the Score Becomes the Target: Rethinking Metric Validity in Autonomous Drivingcited by 0
  26. 2026Verifier Errors in RLVR: Reward Hacking, Limits of Feedback, and Selective Controlcited by 0
  27. 2026Binarization Flattens the Score Spacecited by 0
  28. 2026Revisiting scaling laws for reward optimizationcited by 0
  29. 2026DuoDiscover: Scaling Autonomous Discovery Beyond the Evaluation Bottleneckcited by 0