mlsanityData to 7 Oct 2026

Catastrophic Goodhart: regularizing RLHF with KL divergence does not mitigate heavy-tailed reward misspecification

arXiv 2407.14503 ↗First posted 19 Jul 2024cs.LGcited by 20

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

When applying reinforcement learning from human feedback (RLHF), the reward is learned from data and, therefore, always has some error. It is common to mitigate this by regularizing the policy with KL divergence from a base model, with the hope that balancing reward with regularization will achieve desirable outcomes despite this reward misspecification. We show that when the reward function has light-tailed error, optimal policies under less restrictive KL penalties achieve arbitrarily high utility. However, if error is heavy-tailed, some policies obtain arbitrarily high reward despite achieving no more utility than the base model--a phenomenon we call catastrophic Goodhart. We adapt a discrete optimization method to measure the tails of reward models, finding that they are consistent with light-tailed error. However, the pervasiveness of heavy-tailed distributions in many real-world applications indicates that future sources of RL reward could have heavy-tailed error, increasing the likelihood of reward hacking even with KL regularization.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 12 of its 20 references that are in the map. First posted on arXiv 19 Jul 2024.

Builds on

12 of 20 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024RewardBench: Evaluating Reward Models for Language Modeling20 Mar
  2. 2023The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedback31 Oct
  3. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  4. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  5. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 7

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  3. 2015Trust Region Policy Optimizationcited by 2,503
  4. 2018Maximum a Posteriori Policy Optimisationcited by 318
  5. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  6. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 267
  7. 2021Consequences of Misaligned AIcited by 79

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 17.

20 papers.

Sort:

By area:

  1. 2025Reward Shaping to Mitigate Reward Hacking in RLHFcited by 88
  2. 2025Inference-Time Reward Hacking in Large Language Modelscited by 32
  3. 2026Unifying Stable Optimization and Reference Regularization in RLHFcited by 4
  4. 2026Survive or Collapse: The Asymmetric Roles of Data Gating and Reward Grounding in Self-Play RLcited by 4
  5. 2025The Strong, Weak and Benign Goodhart's law. An independence-free and paradigm-agnostic formalisationcited by 2
  6. 2026Best-of-Tails: Bridging Optimism and Pessimism in Inference-Time Alignmentcited by 2
  7. 2026No Certificate for Alignment: Two Independent Impossibilities and the Pareto Frontier of Achievable Safety Guaranteescited by 1
  8. 2026General Preference Reinforcement Learningcited by 1
  9. 2026Tail-Shape Estimation in LLM Evaluation Is Fragile: A Protocol for Diagnosing False Positivescited by 1
  10. 2026Breaking the Solver Bottleneck: Training Task Generators at the Learnable Frontiercited by 1
  11. 2026Disentangling Deception and Hallucination Failures in LLMscited by 0
  12. 2026MHPO: Modulated Hazard-aware Policy Optimization for Stable Reinforcement Learningcited by 0
8 more
  1. 2026Relative Density Ratio Optimization for Stable and Statistically Consistent Model Alignmentcited by 0
  2. 2026Tail-Aware Information-Theoretic Bounds for LLM Alignment under Heavy-Tailed Rewardscited by 0
  3. 2026Fragility of Value under Imperfect Alignmentcited by 0
  4. 2026Robust Code RL via Faulty-Code-Driven Test case Synthesis and Dense Reward Shapingcited by 0
  5. 2026CHASE: How Content Ecosystems Are Reshaped When Ranking Is the Only Targetcited by 0
  6. 2026Verifier Errors in RLVR: Reward Hacking, Limits of Feedback, and Selective Controlcited by 0
  7. 2026Dual-Channel Robust Group-Relative Policy Optimization via Advantage and Sequence-Weight Estimationcited by 0
  8. 2026TRACE: Diagnosing Verifier Brittleness in Agentic Evaluationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Learning a Pessimistic Reward Model in RLHF
  2. 2026A Unifying Lens on Reward Uncertainty in RLHF
  3. 2025Rethinking KL Regularization in RLHF: From Value Estimation to Gradient Optimization
  4. 2025Logarithmic Regret for Online KL-Regularized Reinforcement Learning
  5. 2024The Perils of Optimizing Learned Reward Functions: Low Training Error Does Not Guarantee Low Regret
  6. 2026f-Divergence Regularized RLHF: Two Tales of Sampling and Unified Analyses
  7. 2026When Errors Can Be Beneficial: A Categorization of Imperfect Rewards for Policy Gradient
  8. 2025Information-Theoretic Reward Modeling for Stable RLHF: Detecting and Mitigating Reward Hacking