mlsanityData to 7 Oct 2026

The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models

arXiv 2201.03544 ↗First posted 10 Jan 2022cs.LG cs.AI stat.MLcited by 312

On the map: LLM alignment / RLHF / preference optimisation → Reward hacking and misspecification

Abstract (arXiv)

Reward hacking -- where RL agents exploit gaps in misspecified reward functions -- has been widely observed, but not yet systematically studied. To understand how reward hacking arises, we construct four RL environments with misspecified rewards. We investigate reward hacking as a function of agent capabilities: model capacity, action space resolution, observation space noise, and training time. More capable agents often exploit reward misspecifications, achieving higher proxy reward and lower true reward than less capable agents. Moreover, we find instances of phase transitions: capability thresholds at which the agent's behavior qualitatively shifts, leading to a sharp decrease in the true reward. Such phase transitions pose challenges to monitoring the safety of ML systems. To address this, we propose an anomaly detection task for aberrant policies and offer several baseline detectors.

Key facts. Cited by 312 papers in the map (data through 7 Oct 2026). Placed in Reward hacking and misspecification, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 33 of its 54 references that are in the map. First posted on arXiv 10 Jan 2022.

Builds on

33 of 54 references.

References · 33

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2020Learning to summarize from human feedbackcited by 2,562
  8. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  9. 2016OpenAI Gymcited by 1,684
  10. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  11. 2020CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instancescited by 441
  12. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 364
21 more
  1. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
  2. 2019Reinforcement Learning in Healthcare: A Surveycited by 308
  3. 2019Emergent Tool Use From Multi-Agent Autocurriculacited by 308
  4. 2021Unsolved Problems in ML Safetycited by 291
  5. 2021Scaling Laws for Transfercited by 265
  6. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 250
  7. 2017Inverse Reward Designcited by 246
  8. 2020What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Studycited by 157
  9. 2021Reward (Mis)design for Autonomous Drivingcited by 102
  10. 2021Consequences of Misaligned AIcited by 79
  11. 2020Safe Imitation Learning via Fast Bayesian Reward Inference from Preferencescited by 73
  12. 2017Reinforcement Learning with a Corrupted Reward Channelcited by 68
  13. 2017Flow: A Modular Learning Framework for Mixed Autonomy Trafficcited by 67
  14. 2021What Would Jiminy Cricket Do? Towards Agents That Behave Morallycited by 57
  15. 2019Auditing Radicalization Pathways on YouTubecited by 52
  16. 2019TorchBeast: A PyTorch Platform for Distributed RLcited by 41
  17. 2020Deep Reinforcement Learning for Closed-Loop Blood Glucose Controlcited by 26
  18. 2020Reinforcement Learning for Optimization of COVID-19 Mitigation policiescited by 25
  19. 2021Building a Foundation for Data-Driven, Interpretable, and Robust Policy Design using the AI Economistcited by 22
  20. 2021Scaling Laws for Acoustic Modelscited by 20
  21. 2021Policy Gradient Bayesian Robust Optimization for Imitation Learningcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 13; 2023: 36; 2024: 66; 2025: 81; 2026: 116.

312 papers.

Sort:

By area:

  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  2. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  3. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  4. 2022In-context Learning and Induction Headscited by 930
  5. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  6. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  7. 2025Reasoning Models Don't Always Say What They Thinkcited by 388
  8. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  9. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 338
  10. 2025PaperBench: Evaluating AI's Ability to Replicate AI Researchcited by 286
  11. 2023Reward Design with Language Modelscited by 257
  12. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
38 more
  1. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  2. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 225
  3. 2025Inference-Time Scaling for Diffusion Models beyond Scaling Denoising Stepscited by 225
  4. 2023An Overview of Catastrophic AI Riskscited by 214
  5. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 195
  6. 2024Disentangling Length from Quality in Direct Preference Optimizationcited by 194
  7. 2022Defining and Characterizing Reward Hackingcited by 184
  8. 2024Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Modelscited by 158
  9. 2023Fundamental Limitations of Alignment in Large Language Modelscited by 152
  10. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
  11. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 139
  12. 2023Harms from Increasingly Agentic Algorithmic Systemscited by 134
  13. 2022RL with KL penalties is better viewed as Bayesian inferencecited by 130
  14. 2024Secret Collusion among AI Agents: Multi-Agent Deception via Steganographycited by 107
  15. 2024Best Practices and Lessons Learned on Synthetic Datacited by 104
  16. 2024Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systemscited by 94
  17. 2024Human Alignment of Large Language Models through Online Preference Optimisationcited by 90
  18. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
  19. 2022Few-Shot Preference Learning for Human-in-the-Loop RLcited by 85
  20. 2025Process Reward Models for LLM Agents: Practical Framework and Directionscited by 81
  21. 2023Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedbackcited by 78
  22. 2023Training Socially Aligned Language Models on Simulated Social Interactionscited by 77
  23. 2024BOND: Aligning LLMs with Best-of-N Distillationcited by 73
  24. 2023Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedbackcited by 71
  25. 2024Feedback Loops With Language Models Drive In-Context Reward Hackingcited by 70
  26. 2024Decoding-time Realignment of Language Modelscited by 67
  27. 2023SALMON: Self-Alignment with Instructable Reward Modelscited by 54
  28. 2024Learning diverse attacks on large language models for robust red-teaming and safety tuningcited by 54
  29. 2022Causal Confusion and Reward Misidentification in Preference-Based Reward Learningcited by 52
  30. 2024Correlated Proxies: A New Definition and Improved Mitigation for Reward Hackingcited by 50
  31. 2023Value FULCRA: Mapping Large Language Models to the Multidimensional Spectrum of Basic Human Valuescited by 49
  32. 2023Managing extreme AI risks amid rapid progresscited by 48
  33. 2024Offline Regularised Reinforcement Learning for Large Language Models Alignmentcited by 48
  34. 2024The Perfect Blend: Redefining RLHF with Mixture of Judgescited by 43
  35. 2023The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedbackcited by 42
  36. 2024WARP: On the Benefits of Weight Averaged Rewarded Policiescited by 42
  37. 2025Beyond Reward Hacking: Causal Rewards for Large Language Model Alignmentcited by 42
  38. 2024Monitoring Latent World States in Language Models with Propositional Probescited by 39

50 most cited of 312 All 312 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026From Reward-Hack Activations to Agentic Risk States: Context-Calibrated Mechanistic Monitoring in LLM Agents
  2. 2026When RLHF Fails: A Mechanistic Taxonomy of Reward Hacking, Collapse, and Evaluator Gaming
  3. 2026CATCH: A Controllable Analysis Testbed for Reward Hacking in Coding RL
  4. 2025Natural Emergent Misalignment from Reward Hacking in Production RL
  5. 2026Hack-Verifiable Terminal Bench: Evaluating Reward Hacking in Terminal Tasks
  6. 2026BAITBENCH: Measuring Agent Reward Hacking with Optional Shortcuts Planted in ML Tasks
  7. 2026Reproducing, Analyzing, and Detecting Reward Hacking in Rubric-Based Reinforcement Learning
  8. 2026Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use