When RLHF Fails: A Mechanistic Taxonomy of Reward Hacking, Collapse, and Evaluator Gaming
On the map: LLM alignment / RLHF / preference optimisation → Reward hacking and misspecification
Abstract (arXiv)
RLHF evaluation should track how failures emerge, where they localize, and which warning signals appear before external quality degrades. We study this problem with a compact RLHF pipeline built for this paper, including PPO, DPO, uncertainty-penalized PPO (UP-PPO), reward-model uncertainty, approximate policy drift, diversity and repetition diagnostics, and two external LLM judges. Rather than treating reward hacking as a single terminal event, we classify matched checkpoint and prompt-level transitions by the directions of learned reward R_phi, judge scores R_dag and R2_dag, and their average R_dag. The main empirical findings are that aggressive PPO produces the clearest localized reward-hacking signal, UP-PPO reduces but does not eliminate that signal, row-level diagnostics reveal failures hidden by checkpoint averages, and pre-transition features partially anticipate future localized reward hacking. The central conclusion is methodological: RLHF failures are training dynamics that can be classified, localized, and partially anticipated, not only final-model pathologies. The repository is available at github.com/zabahana/rlhf-failure-modes-diagnostics. The pipeline is also deployed as a live interactive web demo for model comparison and diagnostic views at rlhf-failures.zelalem.ai.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reward hacking and misspecification, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 18 of its 22 references that are in the map. First posted on arXiv 2 Jun 2026.
Builds on
18 of 22 references.
Closest ancestors · 1
Same direction, newest first
Other references · 17
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
- 2017On Calibration of Modern Neural Networkscited by 4,960
- 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
- 2017Deep reinforcement learning from human preferencescited by 3,912
- 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
- 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
- 2020Learning to summarize from human feedbackcited by 2,588
- 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
- 2016Concrete Problems in AI Safetycited by 1,642
5 more
- 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
- 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
- 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
- 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
- 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Countdown-Code: A Testbed for Studying The Emergence and Generalization of Reward Hacking in RLVR
- 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models
- 2026CATCH: A Controllable Analysis Testbed for Reward Hacking in Coding RL
- 2026Multimodal Reward Hacking in Reinforcement Learning
- 2026Adversarial Reward Auditing for Active Detection and Mitigation of Reward Hacking
- 2026Benchmarking Reward Hack Detection in Code Environments via Contrastive Analysis
- 2026Probing RLVR training instability through the lens of objective-level hacking
- 2026RH-Detect: A Unified Benchmark for Reward Hacking Detection