mlsanityData to 7 Oct 2026

Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective

arXiv 1908.04734 ↗First posted 13 Aug 2019cs.AI cs.LGcited by 107

Not on the map. Why?

Cited by 28 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Can humans get arbitrarily capable reinforcement learning (RL) agents to do their bidding? Or will sufficiently capable RL agents always find ways to bypass their intended objectives by shortcutting their reward signal? This question impacts how far RL can be scaled, and whether alternative paradigms must be developed in order to build safe artificial general intelligence. In this paper, we study when an RL agent has an instrumental goal to tamper with its reward process, and describe design principles that prevent instrumental goals for two different types of reward tampering (reward function tampering and RF-input tampering). Combined, the design principles can prevent both types of reward tampering from being instrumental goals. The analysis benefits from causal influence diagrams to provide intuitive yet precise formalizations.

Key facts. Cited by 107 papers in the map (data through 7 Oct 2026). First posted on arXiv 13 Aug 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 11; 2023: 12; 2024: 10; 2025: 32; 2026: 34.

107 papers.

Sort:

By area:

  1. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  2. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  3. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  4. 2025Reasoning Models Don't Always Say What They Thinkcited by 388
  5. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
  6. 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 236
  7. 2022Defining and Characterizing Reward Hackingcited by 184
  8. 2021Alignment of Language Agentscited by 167
  9. 2024Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Modelscited by 158
  10. 2024Best Practices and Lessons Learned on Synthetic Datacited by 104
  11. 2023Training Socially Aligned Language Models on Simulated Social Interactionscited by 77
  12. 2025SRPO: A Cross-Domain Implementation of Large-Scale Reinforcement Learning on LLMcited by 67
38 more
  1. 2021Agent Incentives: A Causal Perspectivecited by 51
  2. 2024On Targeted Manipulation and Deception when Optimizing LLMs for User Feedbackcited by 45
  3. 2021Visual Adversarial Imitation Learning using Variational Modelscited by 42
  4. 2025Beyond Reward Hacking: Causal Rewards for Large Language Model Alignmentcited by 42
  5. 2023Honesty Is the Best Policy: Defining and Mitigating AI Deceptioncited by 40
  6. 2025S2R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learningcited by 37
  7. 2024AI Alignment with Changing and Influenceable Reward Functionscited by 33
  8. 2024Uncovering Deceptive Tendencies in Language Models: A Simulated Company AI Assistantcited by 32
  9. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 30
  10. 2022Counterfactual harmcited by 29
  11. 2022Reward Gaming in Conditional Text Generationcited by 29
  12. 2022Discovering Agentscited by 28
  13. 2021User Tampering in Reinforcement Learning Recommender Systemscited by 23
  14. 2023Causal Reinforcement Learning: A Surveycited by 22
  15. 2022Path-Specific Objectives for Safer Agent Incentivescited by 20
  16. 2025MONA: Myopic Optimization with Non-myopic Approval Can Mitigate Multi-step Reward Hackingcited by 19
  17. 2025Detecting Proxy Gaming in RL and LLM Alignment via Evaluator Stress Testscited by 19
  18. 2025RLHS: Mitigating Misalignment in RLHF with Hindsight Simulationcited by 15
  19. 2025Think2SQL: Reinforce LLM Reasoning Capabilities for Text2SQLcited by 14
  20. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
  21. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 14
  22. 2026Do Androids Dream of Breaking the Game? Systematically Auditing AI Agent Benchmarks with BenchJackcited by 13
  23. 2023Human Control: Definitions and Algorithmscited by 10
  24. 2023SeMAIL: Eliminating Distractors in Visual Imitation via Separated Modelscited by 10
  25. 2021Equilibrium Refinements for Multi-Agent Influence Diagrams: Theory and Practicecited by 9
  26. 2026Adversarial Reward Auditing for Active Detection and Mitigation of Reward Hackingcited by 9
  27. 2023Designing Fiduciary Artificial Intelligencecited by 8
  28. 2025Investigating CoT Monitorability in Large Reasoning Modelscited by 8
  29. 2024Measuring Goal-Directednesscited by 7
  30. 2025Beyond Accuracy: Dissecting Mathematical Reasoning for LLMs Under Reinforcement Learningcited by 7
  31. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 7
  32. 2025Machine Bullshit: Characterizing the Emergent Disregard for Truth in Large Language Modelscited by 6
  33. 2025IFDECORATOR: Wrapping Instruction Following Reinforcement Learning with Verifiable Rewardscited by 6
  34. 2025Collaborate, Deliberate, Evaluate: How LLM Alignment Affects Coordinated Multi-Agent Outcomescited by 6
  35. 2025Goal Reaching with Eikonal-Constrained Hierarchical Quasimetric Reinforcement Learningcited by 6
  36. 2026Do Agent Benchmarks Measure Capability? Protocol Validity in the Age of Agentic AIcited by 6
  37. 2022Safe Deep RL in 3D Environments using Human Feedbackcited by 5
  38. 2022A Complete Criterion for Value of Information in Soluble Influence Diagramscited by 5

50 most cited of 107 All 107 (JSON)