# Catastrophic Goodhart: regularizing RLHF with KL divergence does not mitigate heavy-tailed reward misspecification

arXiv: https://arxiv.org/abs/2407.14503 · HTML: https://mlsanity.com/paper/2407.14503/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 12 of its 20 references that are in the map.
- First posted on arXiv 19 Jul 2024.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Reward modeling and RLHF](https://mlsanity.com/alignment/reward-modeling-and-rlhf/)

## Builds on (most cited first, 12 of 12)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [Scaling Laws for Reward Model Overoptimization](https://mlsanity.com/paper/2210.10760/) (2022)
- [RewardBench: Evaluating Reward Models for Language Modeling](https://mlsanity.com/paper/2403.13787/) (2024)
- [Maximum a Posteriori Policy Optimisation](https://mlsanity.com/paper/1806.06920/) (2018)
- [Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog](https://mlsanity.com/paper/1907.00456/) (2019)
- [Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents](https://mlsanity.com/paper/1709.06009/) (2017)
- [Consequences of Misaligned AI](https://mlsanity.com/paper/2102.03896/) (2021)
- [The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2311.00168/) (2023)

## Built on by (most cited first, 23 of 23)

- [Reward Shaping to Mitigate Reward Hacking in RLHF](https://mlsanity.com/paper/2502.18770/) (2025)
- [Inference-Time Reward Hacking in Large Language Models](https://mlsanity.com/paper/2506.19248/) (2025)
- [Unifying Stable Optimization and Reference Regularization in RLHF](https://mlsanity.com/paper/2602.11523/) (2026)
- [Survive or Collapse: The Asymmetric Roles of Data Gating and Reward Grounding in Self-Play RL](https://mlsanity.com/paper/2605.22217/) (2026)
- [Best-of-Tails: Bridging Optimism and Pessimism in Inference-Time Alignment](https://mlsanity.com/paper/2603.06797/) (2026)
- [Breaking the Solver Bottleneck: Training Task Generators at the Learnable Frontier](https://mlsanity.com/paper/2606.18284/) (2026)
- [The Strong, Weak and Benign Goodhart's law. An independence-free and paradigm-agnostic formalisation](https://mlsanity.com/paper/2505.23445/) (2025)
- [No Certificate for Alignment: Two Independent Impossibilities and the Pareto Frontier of Achievable Safety Guarantees](https://mlsanity.com/paper/2603.08761/) (2026)
- [General Preference Reinforcement Learning](https://mlsanity.com/paper/2605.18721/) (2026)
- [Tail-Shape Estimation in LLM Evaluation Is Fragile: A Protocol for Diagnosing False Positives](https://mlsanity.com/paper/2606.16511/) (2026)
- [Disentangling Deception and Hallucination Failures in LLMs](https://mlsanity.com/paper/2602.14529/) (2026)
- [MHPO: Modulated Hazard-aware Policy Optimization for Stable Reinforcement Learning](https://mlsanity.com/paper/2603.16929/) (2026)
- [Relative Density Ratio Optimization for Stable and Statistically Consistent Model Alignment](https://mlsanity.com/paper/2604.04410/) (2026)
- [Tail-Aware Information-Theoretic Bounds for LLM Alignment under Heavy-Tailed Rewards](https://mlsanity.com/paper/2604.10727/) (2026)
- [Fragility of Value under Imperfect Alignment](https://mlsanity.com/paper/2607.28881/) (2026)
- [Robust Code RL via Faulty-Code-Driven Test case Synthesis and Dense Reward Shaping](https://mlsanity.com/paper/2608.24135/) (2026)
- [CHASE: How Content Ecosystems Are Reshaped When Ranking Is the Only Target](https://mlsanity.com/paper/2608.30466/) (2026)
- [Verifier Errors in RLVR: Reward Hacking, Limits of Feedback, and Selective Control](https://mlsanity.com/paper/2609.35677/) (2026)
- [Dual-Channel Robust Group-Relative Policy Optimization via Advantage and Sequence-Weight Estimation](https://mlsanity.com/paper/2609.36944/) (2026)
- [Reinforcement Learning on the Discrete Composition Channel of a Crystal Generator: Validated Gains and Reward Hacking](https://mlsanity.com/paper/2610.03880/) (2026)
- [TRACE: Diagnosing Verifier Brittleness in Agentic Evaluation](https://mlsanity.com/paper/2610.11678/) (2026)
- [Copying Is Not Grounding: Training Faithful Long-Form RAG with Authored-Sentence Credit](https://mlsanity.com/paper/2610.18442/) (2026)
- [Tail Competition Explains Scaling in Best-of-N Sampling for Verifiable Problems](https://mlsanity.com/paper/2610.19362/) (2026)

## Abstract (arXiv)

When applying reinforcement learning from human feedback (RLHF), the reward is learned from data and, therefore, always has some error. It is common to mitigate this by regularizing the policy with KL divergence from a base model, with the hope that balancing reward with regularization will achieve desirable outcomes despite this reward misspecification. We show that when the reward function has light-tailed error, optimal policies under less restrictive KL penalties achieve arbitrarily high utility. However, if error is heavy-tailed, some policies obtain arbitrarily high reward despite achieving no more utility than the base model--a phenomenon we call catastrophic Goodhart. We adapt a discrete optimization method to measure the tails of reward models, finding that they are consistent with light-tailed error. However, the pervasiveness of heavy-tailed distributions in many real-world applications indicates that future sources of RL reward could have heavy-tailed error, increasing the likelihood of reward hacking even with KL regularization.