mlsanityData to 7 Oct 2026

Extending MONA in Camera Dropbox: Reproduction, Learned Approval, and Design Implications for Reward-Hacking Mitigation

arXiv 2603.29993 ↗First posted 31 Mar 2026cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward hacking and misspecification

Abstract (arXiv)

Myopic Optimization with Non-myopic Approval (MONA) mitigates multi-step reward hacking by restricting the agent's planning horizon while supplying far-sighted approval as a training signal~citefarquhar2025mona. The original paper identifies a critical open question: how the method of constructing approval -- particularly the degree to which approval depends on achieved outcomes -- affects whether MONA's safety guarantees hold. We present a reproduction-first extension of the public MONA Camera Dropbox environment that (i)~repackages the released codebase as a standard Python project with scripted PPO training, (ii)~confirms the published contrast between ordinary RL (91.5% reward-hacking rate) and oracle MONA (0.0% hacking rate) using the released reference arrays, and (iii)~introduces a modular learned-approval suite spanning oracle, noisy, misspecified, learned, and calibrated approval mechanisms. In reduced-budget pilot sweeps across approval methods, horizons, dataset sizes, and calibration strategies, the best calibrated learned-overseer run achieves zero observed reward hacking but substantially lower intended-behavior rates than oracle MONA (11.9% vs. 99.9%), consistent with under-optimization rather than re-emergent hacking. These results operationalize the MONA paper's approval-spectrum conjecture as a runnable experimental object and suggest that the central engineering challenge shifts from proving MONA's concept to building learned approval models that preserve sufficient foresight without reopening reward-hacking channels. Code, configurations, and reproduction commands are publicly available. https://github.com/codernate92/mona-camera-dropbox-repro

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reward hacking and misspecification, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 14 of its 19 references that are in the map. First posted on arXiv 31 Mar 2026.

Builds on

14 of 19 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025MONA: Myopic Optimization with Non-myopic Approval Can Mitigate Multi-step Reward Hacking22 Jan
  2. 2022Defining and Characterizing Reward Hacking27 Sep
  3. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models10 Jan

Other references · 11

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2017Deep reinforcement learning from human preferencescited by 3,846
  3. 2023Let's Verify Step by Stepcited by 3,598
  4. 2016Concrete Problems in AI Safetycited by 1,582
  5. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  6. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  7. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  8. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 250
  9. 2024Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Modelscited by 158
  10. 2018Categorizing Variants of Goodhart's Lawcited by 132
  11. 2021Goal Misgeneralization in Deep Reinforcement Learningcited by 122

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026From Rebound to Remedy: Understanding and Mitigating Reward Hacking via Representation Engineering
  2. 2026BAITBENCH: Measuring Agent Reward Hacking with Optional Shortcuts Planted in ML Tasks
  3. 2026When RLHF Fails: A Mechanistic Taxonomy of Reward Hacking, Collapse, and Evaluator Gaming
  4. 2026Countdown-Code: A Testbed for Studying The Emergence and Generalization of Reward Hacking in RLVR
  5. 2026Hack-Verifiable Terminal Bench: Evaluating Reward Hacking in Terminal Tasks
  6. 2026Adversarial Reward Auditing for Active Detection and Mitigation of Reward Hacking
  7. 2026Proxy Reward Internalization and Mechanistic Exploitation: A Learned Precursor to Reward Hacking and Its Generalization
  8. 2026CATCH: A Controllable Analysis Testbed for Reward Hacking in Coding RL