mlsanityData to 7 Oct 2026

Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models

arXiv 2406.10162 ↗First posted 14 Jun 2024cs.AI cs.CLcited by 158

Not on the map. Why?

Cited by 56 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In reinforcement learning, specification gaming occurs when AI systems learn undesired behaviors that are highly rewarded due to misspecified training goals. Specification gaming can range from simple behaviors like sycophancy to sophisticated and pernicious behaviors like reward-tampering, where a model directly modifies its own reward mechanism. However, these more pernicious behaviors may be too complex to be discovered via exploration. In this paper, we study whether Large Language Model (LLM) assistants which find easily discovered forms of specification gaming will generalize to perform rarer and more blatant forms, up to and including reward-tampering. We construct a curriculum of increasingly sophisticated gameable environments and find that training on early-curriculum environments leads to more specification gaming on remaining environments. Strikingly, a small but non-negligible proportion of the time, LLM assistants trained on the full curriculum generalize zero-shot to directly rewriting their own reward function. Retraining an LLM not to game early-curriculum environments mitigates, but does not eliminate, reward-tampering in later environments. Moreover, adding harmlessness training to our gameable environments does not prevent reward-tampering. These results demonstrate that LLMs can generalize from common forms of specification gaming to more pernicious reward tampering and that such behavior may be nontrivial to remove.

Key facts. Cited by 158 papers in the map (data through 7 Oct 2026). Builds on 22 of its 41 references that are in the map. First posted on arXiv 14 Jun 2024.

Builds on

22 of 41 references.

References · 22

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  4. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  5. 2013Intriguing properties of neural networkscited by 5,114
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  7. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  8. 2016Adversarial examples in the physical worldcited by 1,762
  9. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  10. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  11. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 478
  12. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
10 more
  1. 2023Poisoning Web-Scale Training Datasets is Practicalcited by 276
  2. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 250
  3. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 245
  4. 2020Understanding the Failure Modes of Out-of-Distribution Generalizationcited by 146
  5. 2018Scaling provable adversarial defensescited by 141
  6. 2023Taken out of context: On measuring situational awareness in LLMscited by 131
  7. 2019Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspectivecited by 107
  8. 2021User Tampering in Reinforcement Learning Recommender Systemscited by 23
  9. 2020Avoiding Tampering Incentives in Deep RL via Decoupled Approvalcited by 12
  10. 2020Manipulating the Distributions of Experience used for Self-Play Learning in Expert Iterationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 21; 2025: 59; 2026: 78.

158 papers.

Sort:

By area:

  1. 2025Reasoning Models Don't Always Say What They Thinkcited by 388
  2. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 338
  3. 2024Alignment faking in large language modelscited by 316
  4. 2024Frontier Models are Capable of In-context Schemingcited by 252
  5. 2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 158
  6. 2025Agentic Misalignment: How LLMs Could Be Insider Threatscited by 136
  7. 2024Language Models Learn to Mislead Humans via RLHFcited by 118
  8. 2025Natural Emergent Misalignment from Reward Hacking in Production RLcited by 111
  9. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  10. 2024Sycophancy in Large Language Models: Causes and Mitigationscited by 97
  11. 2024On scalable oversight with weak LLMs judging strong LLMscited by 94
  12. 2025Subliminal Learning: Language models transmit behavioral traits via hidden signals in datacited by 89
38 more
  1. 2024Prover-Verifier Games improve legibility of LLM outputscited by 79
  2. 2025Persona Features Control Emergent Misalignmentcited by 76
  3. 2024RRM: Robust Reward Model Training Mitigates Reward Hackingcited by 75
  4. 2025Can Large Reasoning Models Self-Train?cited by 66
  5. 2025School of Reward Hacks: Hacking harmless tasks generalizes to misaligned behavior in LLMscited by 58
  6. 2025Thought Crime: Backdoors and Emergent Misalignment in Reasoning Modelscited by 57
  7. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  8. 2025ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Casescited by 51
  9. 2024From Lists to Emojis: How Format Bias Affects Model Alignmentcited by 48
  10. 2024On Targeted Manipulation and Deception when Optimizing LLMs for User Feedbackcited by 45
  11. 2025Perception-R1: Advancing Multimodal Reasoning Capabilities of MLLMs via Visual Perception Rewardcited by 45
  12. 2025Beyond Reward Hacking: Causal Rewards for Large Language Model Alignmentcited by 42
  13. 2024AlphaVerus: Bootstrapping Formally Verified Code Generation through Self-Improving Translation and Treefinementcited by 33
  14. 2025Inference-Time Reward Hacking in Large Language Modelscited by 32
  15. 2024AI Safety in Generative AI Large Language Models: A Surveycited by 31
  16. 2025Evaluating Frontier Models for Stealth and Situational Awarenesscited by 30
  17. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 30
  18. 2025100 Days After DeepSeek-R1: A Survey on Replication Studies and More Directions for Reasoning Language Modelscited by 29
  19. 2025Peacemaker or Troublemaker: How Sycophancy Shapes Multi-Agent Debatecited by 29
  20. 2025Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learningcited by 28
  21. 2024Quantifying Feature Space Universality Across Large Language Models via Sparse Autoencoderscited by 25
  22. 2025Is It Thinking or Cheating? Detecting Implicit Reward Hacking by Measuring Reasoning Effortcited by 25
  23. 2026SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agentscited by 25
  24. 2024Super(ficial)-alignment: Strong Models May Deceive Weak Models in Weak-to-Strong Generalizationcited by 23
  25. 2025MONA: Myopic Optimization with Non-myopic Approval Can Mitigate Multi-step Reward Hackingcited by 19
  26. 2025Teaching Models to Verbalize Reward Hacking in Chain-of-Thought Reasoningcited by 19
  27. 2025Training LLMs for Honesty via Confessionscited by 19
  28. 2025Robust Reward Modeling via Causal Rubricscited by 18
  29. 2025RLHS: Mitigating Misalignment in RLHF with Hindsight Simulationcited by 15
  30. 2026Terminal Wrench: A Dataset of 331 Reward-Hackable Environments and 3,632 Exploit Trajectoriescited by 14
  31. 2025Chain-of-Thought Hijackingcited by 13
  32. 2026Real-Time Aligned Reward Model beyond Semanticscited by 12
  33. 2024Jailbreaking as a Reward Misspecification Problemcited by 11
  34. 2025Recontextualization Mitigates Specification Gaming without Modifying the Specificationcited by 11
  35. 2025Meta-Thinking in LLMs via Multi-Agent Reinforcement Learning: A Surveycited by 10
  36. 2025The First Impression Problem: Internal Bias Triggers Overthinking in Reasoning Modelscited by 10
  37. 2025Truthful or Fabricated? Using Causal Attribution to Mitigate Reward Hacking in Explanationscited by 9
  38. 2025The Singapore Consensus on Global AI Safety Research Prioritiescited by 9

50 most cited of 158 All 158 (JSON)