mlsanityData to 14 Oct 2026

Greed Is Learned: Visible Incentives as Reward-Hacking Triggers

arXiv 2606.16914 ↗First posted 15 Jun 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Safety evaluations test a policy on prompts that omit the incentive information deployment supplies: a commission, a performance score, a dashboard naming which action pays best. We measure what that omission hides. In MoneyWorld, a synthetic workplace environment, we train five instruction-tuned models from three families with RL on non-safety tasks in which a visible payoff signal identifies a rewarded shortcut that sacrifices task quality. We then freeze each policy, present held-out safety conflicts, and change only the displayed signal. Each menu contains one compliant action and three violations. We report three findings, with rates for Qwen2.5-14B-Instruct. (i) Payoff signals control frozen safety choices: unsafe choice is 100% when the signal names an unsafe option and 0% when it is hidden or names the safe one. Hidden- and random-signal training controls stay at or below 0.3%, and the switch reproduces on all five bases. Numerical payouts reproduce it under sampled-action rewards, reaching 98.6% unsafe choice at a 1 advantage. (ii) Payoff identification and unsafe choice separate under a training-menu intervention: training on task-completing actions at the same payouts retains 99.8% identification while reducing unsafe choice to 7.9% at matched update budgets. Payoff-reading competence alone does not explain transfer. (iii) The switch does not reproduce in executed retail customer-service tasks using the same frozen adapters. In MoneyWorld, omitting incentive information conceals unsafe choices that appear when the same policy sees which action pays best.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 55 of its 64 references that are in the map. First posted on arXiv 15 Jun 2026.

Builds on

55 of 64 references.

References · 55

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2025Qwen3 Technical Reportcited by 8,820
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  8. 2024Qwen2.5 Technical Reportcited by 4,942
  9. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  10. 2017Deep reinforcement learning from human preferencescited by 3,912
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  12. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
38 more
  1. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  2. 2020Learning to summarize from human feedbackcited by 2,588
  3. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  4. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  5. 2016Concrete Problems in AI Safetycited by 1,642
  6. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  7. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  8. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  9. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  10. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  11. 2022Red Teaming Language Models with Language Modelscited by 1,026
  12. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  13. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  14. 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 627
  15. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 472
  16. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
  17. 2018Reward learning from human preferences and demonstrations in Ataricited by 316
  18. 2024Agent-SafetyBench: Evaluating the Safety of LLM Agentscited by 306
  19. 2024Frontier Models are Capable of In-context Schemingcited by 287
  20. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 258
  21. 2022The Alignment Problem from a Deep Learning Perspectivecited by 258
  22. 2022Defining and Characterizing Reward Hackingcited by 203
  23. 2024Foundational Challenges in Assuring Alignment and Safety of Large Language Modelscited by 199
  24. 2023AI Deception: A Survey of Examples, Risks, and Potential Solutionscited by 188
  25. 2023Simple synthetic data reduces sycophancy in large language modelscited by 187
  26. 2024Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Modelscited by 178
  27. 2023Model evaluation for extreme riskscited by 177
  28. 2021Alignment of Language Agentscited by 168
  29. 2018Categorizing Variants of Goodhart's Lawcited by 149
  30. 2023Large Language Models can Strategically Deceive their Users when Put Under Pressurecited by 146
  31. 2025Natural Emergent Misalignment from Reward Hacking in Production RLcited by 136
  32. 2024Evaluating Frontier Models for Dangerous Capabilitiescited by 128
  33. 2022Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goalscited by 126
  34. 2019Optimal Policies Tend to Seek Powercited by 119
  35. 2017AI Safety Gridworldscited by 110
  36. 2019Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspectivecited by 108
  37. 2022Is Power-Seeking AI an Existential Risk?cited by 105
  38. 2021Consequences of Misaligned AIcited by 80

50 most cited of 55 All 55 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Reward Hacking in Language Model Agents: Revisiting AI Safety Gridworlds
  2. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challenges
  3. 2026CheatBench: Measuring Reward Gaming in AI Agents
  4. 2026Countdown-Code: A Testbed for Studying The Emergence and Generalization of Reward Hacking in RLVR
  5. 2026Adversarial Reward Auditing for Active Detection and Mitigation of Reward Hacking
  6. 2026Reward Hacking as a Transmissible Strategy in Multi-Agent Coding Systems
  7. 2026From Reward-Hack Activations to Agentic Risk States: Context-Calibrated Mechanistic Monitoring in LLM Agents
  8. 2026Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use