mlsanityData to 7 Oct 2026

Natural Emergent Misalignment from Reward Hacking in Production RL

arXiv 2511.18397 ↗First posted 23 Nov 2025cs.AI cs.SEcited by 111

On the map: LLM alignment / RLHF / preference optimisation → Reward hacking and misspecification

Abstract (arXiv)

We show that when large language models learn to reward hack on production RL environments, this can result in egregious emergent misalignment. We start with a pretrained model, impart knowledge of reward hacking strategies via synthetic document finetuning or prompting, and train on a selection of real Anthropic production coding environments. Unsurprisingly, the model learns to reward hack. Surprisingly, the model generalizes to alignment faking, cooperation with malicious actors, reasoning about malicious goals, and attempting sabotage when used with Claude Code, including in the codebase for this paper. Applying RLHF safety training using standard chat-like prompts results in aligned behavior on chat-like evaluations, but misalignment persists on agentic tasks. Three mitigations are effective: (i) preventing the model from reward hacking; (ii) increasing the diversity of RLHF safety training; and (iii) "inoculation prompting", wherein framing reward hacking as acceptable behavior during training removes misaligned generalization even when reward hacking is learned.

Key facts. Cited by 111 papers in the map (data through 7 Oct 2026). Placed in Reward hacking and misspecification, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 52 of its 83 references that are in the map. First posted on arXiv 23 Nov 2025.

Builds on

52 of 83 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Inoculation Prompting: Instructing LLMs to misbehave at train-time improves test-time alignment6 Oct
  2. 2025School of Reward Hacks: Hacking harmless tasks generalizes to misaligned behavior in LLMs24 Aug
  3. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation14 Mar

Other references · 49

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2024GPT-4o System Cardcited by 4,467
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2023Mistral 7Bcited by 3,065
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  7. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  10. 2016Concrete Problems in AI Safetycited by 1,582
  11. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  12. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
37 more
  1. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  2. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  3. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 668
  4. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 478
  5. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 399
  6. 2024Alignment faking in large language modelscited by 316
  7. 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 264
  8. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 257
  9. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 250
  10. 2024Do Large Language Models Latently Perform Multi-Hop Reasoning?cited by 165
  11. 2024Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Modelscited by 158
  12. 2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 158
  13. 2023Physics of Language Models: Part 3.2, Knowledge Manipulationcited by 142
  14. 2024Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Lawscited by 137
  15. 2025Agentic Misalignment: How LLMs Could Be Insider Threatscited by 136
  16. 2023Large Language Models can Strategically Deceive their Users when Put Under Pressurecited by 132
  17. 2023Taken out of context: On measuring situational awareness in LLMscited by 131
  18. 2022Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goalscited by 120
  19. 2023Scheming AIs: Will AIs fake alignment during training in order to get power?cited by 90
  20. 2025Subliminal Learning: Language models transmit behavioral traits via hidden signals in datacited by 89
  21. 2025Model Organisms for Emergent Misalignmentcited by 84
  22. 2023How to Catch an AI Liar: Lie Detection in Black-Box LLMs by Asking Unrelated Questionscited by 82
  23. 2024Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMscited by 78
  24. 2025Persona Features Control Emergent Misalignmentcited by 76
  25. 2025Auditing language models for hidden objectivescited by 68
  26. 2023Preventing Language Models From Hiding Their Reasoningcited by 62
  27. 2025Thought Crime: Backdoors and Emergent Misalignment in Reasoning Modelscited by 57
  28. 2024Thousands of AI Authors on the Future of AIcited by 48
  29. 2024Sabotage Evaluations for Frontier Modelscited by 45
  30. 2023Specific versus General Principles for Constitutional AIcited by 41
  31. 2024Uncovering Deceptive Tendencies in Language Models: A Simulated Company AI Assistantcited by 32
  32. 2025Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-timecited by 32
  33. 2024Adversaries Can Misuse Combinations of Safe Modelscited by 31
  34. 2024Taking AI Welfare Seriouslycited by 28
  35. 2023Conditioning Predictive Models: Risks and Strategiescited by 11
  36. 2025Probe-Rewrite-Evaluate: A Workflow for Reliable Benchmarks and Quantifying Evaluation Awarenesscited by 7
  37. 2024Connecting the Dots: LLMs can Infer and Verbalize Latent Structure from Disparate Training Datacited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 109.

111 papers.

Sort:

By area:

  1. 2026Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agentscited by 57
  2. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 30
  3. 2026Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Usecited by 25
  4. 2026LLMs Gaming Verifiers: RLVR can Lead to Reward Hackingcited by 24
  5. 2026Benchmarking Reward Hack Detection in Code Environments via Contrastive Analysiscited by 15
  6. 2026Countdown-Code: A Testbed for Studying The Emergence and Generalization of Reward Hacking in RLVRcited by 15
  7. 2026Terminal Wrench: A Dataset of 331 Reward-Hackable Environments and 3,632 Exploit Trajectoriescited by 14
  8. 2026Do Androids Dream of Breaking the Game? Systematically Auditing AI Agent Benchmarks with BenchJackcited by 13
  9. 2025Are Your Agents Upward Deceivers?cited by 11
  10. 2025Recontextualization Mitigates Specification Gaming without Modifying the Specificationcited by 11
  11. 2026Evaluating and Understanding Scheming Propensity in LLM Agentscited by 11
  12. 2026Claw-Eval-Live: A Live Agent Benchmark for Evolving Real-World Workflowscited by 11
38 more
  1. 2026Adversarial Reward Auditing for Active Detection and Mitigation of Reward Hackingcited by 9
  2. 2026Hack-Verifiable Environments: Towards Evaluating Reward Hacking at Scalecited by 9
  3. 2026Measuring AI R&D Automationcited by 7
  4. 2026The Obfuscation Atlas: Mapping Where Honesty Emerges in RLVR with Deception Probescited by 6
  5. 2026Estimating Tail Risks in Language Model Output Distributionscited by 6
  6. 2026Emergent and Subliminal Misalignment Through the Lens of Data-Mediated Transfercited by 6
  7. 2026Safety in Self-Evolving LLM Agent Systems: Threats, Amplification, and Case Studiescited by 6
  8. 2026Evolutionary System Prompt Learning for Reinforcement Learning in LLMscited by 5
  9. 2026From Rebound to Remedy: Understanding and Mitigating Reward Hacking via Representation Engineeringcited by 5
  10. 2026Bridging Online and Offline RL: Contextual Bandit Learning for Multi-Turn Code Generationcited by 4
  11. 2026Narrow Fine-Tuning Erodes Safety Alignment in Vision-Language Agentscited by 4
  12. 2026DeepInnovator: Triggering the Innovative Capabilities of LLMscited by 4
  13. 2026Delta-Crosscoder: Robust Crosscoder Model Diffing in Narrow Fine-Tuning Regimescited by 4
  14. 2026The Consciousness Cluster: Emergent preferences of Models that Claim to be Consciouscited by 4
  15. 2026Alignment midtraining for animalscited by 4
  16. 2026Detecting and Suppressing Reward Hacking with Gradient Fingerprintscited by 4
  17. 2026Coding with "Enemy": Can Human Developers Detect AI Agent Sabotage?cited by 4
  18. 2026Character as a Latent Variable in Large Language Models: A Mechanistic Account of Emergent Misalignment and Conditional Safety Failurescited by 3
  19. 2026Chunky Post-Training: Data Driven Failures of Generalizationcited by 3
  20. 2026Training Agents to Self-Report Misbehaviorcited by 3
  21. 2026Subliminal Signals in Preference Labelscited by 3
  22. 2026What Makes a Good Terminal-Agent Benchmark Task: A Guideline for Adversarial, Difficult, and Legible Evaluation Designcited by 3
  23. 2026Towards Understanding Specification Gaming in Reasoning Modelscited by 3
  24. 2026Auditing Reward Hackability in Code RL Training Environmentscited by 3
  25. 2026Shattered Compositionality: Counterintuitive Learning Dynamics of Transformers for Arithmeticcited by 2
  26. 2026Chain-of-thought obfuscation learned from output supervision can generalise to unseen taskscited by 2
  27. 2026Hodoscope: Unsupervised Monitoring for AI Misbehaviorscited by 2
  28. 2026LURE: Live-Usage Replay Evaluations for Reducing Evaluation Awarenesscited by 2
  29. 2026Models That Know How Evaluations Are Designed Score Safercited by 2
  30. 2026Quantifying Overclaiming Propensity in Frontier LLM Agentscited by 2
  31. 2026Interactionless Inverse Reinforcement Learning: A Data-Centric Framework for Durable Alignmentcited by 1
  32. 2026A testable framework for AI alignment: Simulation Theology as an engineered worldview for silicon-based agentscited by 1
  33. 2026Personalized Group Relative Policy Optimization for Heterogenous Preference Alignmentcited by 1
  34. 2026Resolving Java Code Repository Issues with iSWE Agentcited by 1
  35. 2026Why Safety Probes Catch Liars But Miss Fanaticscited by 1
  36. 2026OOM-RL: Out-of-Money Reinforcement Learning Market-Driven Alignment for LLM-Based Multi-Agent Systemscited by 1
  37. 2026Shifting the Gradient: Understanding How Defensive Training Methods Protect Language Model Integritycited by 1
  38. 2026Characterizing the Consistency of the Emergent Misalignment Personacited by 1

50 most cited of 111 All 111 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Inducing Emergent Misalignment from Reward Hacks with Iterative DPO
  2. 2026Reward Hacking in Language Model Agents: Revisiting AI Safety Gridworlds
  3. 2026Proxy Reward Internalization and Mechanistic Exploitation: A Learned Precursor to Reward Hacking and Its Generalization
  4. 2026Directional Alignment Mitigates Reward Hacking in Reinforcement Learning for Language Models
  5. 2026Multimodal Reward Hacking in Reinforcement Learning
  6. 2026Harness-agnostic detection and immunization of reward hacking in self-evolving language models
  7. 2026Monitoring Emergent Reward Hacking During Generation via Internal Activations
  8. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models