mlsanityData to 7 Oct 2026

MONA: Myopic Optimization with Non-myopic Approval Can Mitigate Multi-step Reward Hacking

arXiv 2501.13011 ↗First posted 22 Jan 2025cs.LG cs.AIcited by 19

On the map: LLM alignment / RLHF / preference optimisation → Reward hacking and misspecification

Abstract (arXiv)

Future advanced AI systems may learn sophisticated strategies through reinforcement learning (RL) that humans cannot understand well enough to safely evaluate. We propose a training method which avoids agents learning undesired multi-step plans that receive high reward (multi-step "reward hacks") even if humans are not able to detect that the behaviour is undesired. The method, Myopic Optimization with Non-myopic Approval (MONA), works by combining short-sighted optimization with far-sighted reward. We demonstrate that MONA can prevent multi-step reward hacking that ordinary RL causes, even without being able to detect the reward hacking and without any extra information that ordinary RL does not get access to. We study MONA empirically in three settings which model different misalignment failure modes including 2-step environments with LLMs representing delegated oversight and encoded reasoning and longer-horizon gridworld environments representing sensor tampering.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in Reward hacking and misspecification, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 36 of its 80 references that are in the map. First posted on arXiv 22 Jan 2025.

Builds on

36 of 80 references.

References · 36

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2017Deep reinforcement learning from human preferencescited by 3,846
  3. 2023Let's Verify Step by Stepcited by 3,598
  4. 2021Program Synthesis with Large Language Modelscited by 3,509
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  7. 2016Concrete Problems in AI Safetycited by 1,582
  8. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  9. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  10. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  11. 2023A Watermark for Large Language Modelscited by 667
  12. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
24 more
  1. 2023AI Control: Improving Safety Despite Intentional Subversioncited by 207
  2. 2024AlphaMath Almost Zero: Process Supervision without Processcited by 160
  3. 2024Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Modelscited by 158
  4. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 155
  5. 2024Language Models Learn to Mislead Humans via RLHFcited by 118
  6. 2019Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspectivecited by 107
  7. 2024GLoRe: When, Where, and How to Improve LLM Reasoning via Global and Local Refinementscited by 101
  8. 2017Reinforcement Learning with a Corrupted Reward Channelcited by 68
  9. 2024Multi-turn Reinforcement Learning from Preference Human Feedbackcited by 65
  10. 2023Preventing Language Models From Hiding Their Reasoningcited by 62
  11. 2024Hidden in Plain Text: Emergence & Mitigation of Steganographic Collusion in LLMscited by 52
  12. 2021Agent Incentives: A Causal Perspectivecited by 51
  13. 2020Neurosymbolic Reinforcement Learning with Formally Verified Explorationcited by 49
  14. 2024On Targeted Manipulation and Deception when Optimizing LLMs for User Feedbackcited by 45
  15. 2021Heuristic-Guided Reinforcement Learningcited by 41
  16. 2022Estimating and Penalizing Induced Preference Shifts in Recommender Systemscited by 29
  17. 2022Controlling Commercial Cooling Systems Using Reinforcement Learningcited by 25
  18. 2015Early Stopping is Nonparametric Variational Inferencecited by 23
  19. 2021User Tampering in Reinforcement Learning Recommender Systemscited by 23
  20. 2022Path-Specific Objectives for Safer Agent Incentivescited by 20
  21. 2023Learning Optimal Advantage from Preferences and Mistaking it for Rewardcited by 18
  22. 2016Avoiding Wireheading with Value Reinforcement Learningcited by 15
  23. 2025RLHS: Mitigating Misalignment in RLHF with Hindsight Simulationcited by 15
  24. 2020Avoiding Tampering Incentives in Deep RL via Decoupled Approvalcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 9; 2026: 10.

19 papers.

Sort:

By area:

  1. 2025When Chain of Thought is Necessary, Language Models Struggle to Evade Monitorscited by 81
  2. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 30
  3. 2025A Survey on Autonomy-Induced Security Risks in Large Model-Based Agentscited by 28
  4. 2025Teaching Models to Verbalize Reward Hacking in Chain-of-Thought Reasoningcited by 19
  5. 2025Can Reasoning Models Obfuscate Reasoning? Stress-Testing Chain-of-Thought Monitorabilitycited by 18
  6. 2025General agents contain world modelscited by 14
  7. 2025Environment Scaling for Interactive Agentic Experience Collection: A Surveycited by 13
  8. 2025Difficulties with Evaluating a Deception Detector for AIscited by 11
  9. 2026Analyzing and Improving Chain-of-Thought Monitorability Through Information Theorycited by 7
  10. 2026Quantifying the Necessity of Chain of Thought through Opaque Serial Depthcited by 7
  11. 2026A Decision-Theoretic Formalisation of Steganography With Applications to LLM Monitoringcited by 6
  12. 2026From Plan to Action: How Well Do Agents Follow the Plan?cited by 4
7 more
  1. 2026Towards Understanding Specification Gaming in Reasoning Modelscited by 3
  2. 2026Likelihood Hacking in Probabilistic Program Synthesiscited by 2
  3. 2025Higher-Order Belief in Incomplete Information MAIDscited by 1
  4. 2026Draining the Energy Commons: Self-Defeating Over-Appropriation as a Coordination Failure in Agentic LLM Collectivescited by 1
  5. 2025Corrigibility Transformation: Constructing Goals That Accept Updatescited by 0
  6. 2026NEST: Nascent Encoded Steganographic Thoughtscited by 0
  7. 2026Extending MONA in Camera Dropbox: Reproduction, Learned Approval, and Design Implications for Reward-Hacking Mitigationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Reward Hacking in Language Model Agents: Revisiting AI Safety Gridworlds
  2. 2026Countdown-Code: A Testbed for Studying The Emergence and Generalization of Reward Hacking in RLVR
  3. 2024Correlated Proxies: A New Definition and Improved Mitigation for Reward Hacking
  4. 2026Reward Hacking as Equilibrium under Finite Evaluation
  5. 2022Defining and Characterizing Reward Hacking
  6. 2026Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use
  7. 2026From Rebound to Remedy: Understanding and Mitigating Reward Hacking via Representation Engineering
  8. 2025The Horcrux: Mechanistically Interpretable Task Decomposition for Detecting and Mitigating Reward Hacking in Embodied AI Systems