# Golden Handcuffs make safer AI agents

arXiv: https://arxiv.org/abs/2604.13609 · HTML: https://mlsanity.com/paper/2604.13609/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 49 references that are in the map.
- First posted on arXiv 15 Apr 2026.

## Builds on (most cited first, 30 of 30)

- [Towards Out-Of-Distribution Generalization: A Survey](https://mlsanity.com/paper/2108.13624/) (2021)
- [Predict Responsibly: Improving Fairness and Accuracy by Learning to Defer](https://mlsanity.com/paper/1711.06664/) (2017)
- [Reward Model Ensembles Help Mitigate Overoptimization](https://mlsanity.com/paper/2310.02743/) (2023)
- [Optimal Policies Tend to Seek Power](https://mlsanity.com/paper/1912.01683/) (2019)
- [Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective](https://mlsanity.com/paper/1908.04734/) (2019)
- [Trial without Error: Towards Safe Reinforcement Learning via Human Intervention](https://mlsanity.com/paper/1707.05173/) (2017)
- [The Off-Switch Game](https://mlsanity.com/paper/1611.08219/) (2016)
- [Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems](https://mlsanity.com/paper/2405.06624/) (2024)
- [OpenClaw-RL: Train Any Agent Simply by Talking](https://mlsanity.com/paper/2603.10165/) (2026)
- [Non-Vacuous Generalization Bounds for Large Language Models](https://mlsanity.com/paper/2312.17173/) (2023)
- [DOPE: Doubly Optimistic and Pessimistic Exploration for Safe Reinforcement Learning](https://mlsanity.com/paper/2112.00885/) (2021)
- [Deep neural networks have an inbuilt Occam's razor](https://mlsanity.com/paper/2304.06670/) (2023)
- [Learning to Defer with Limited Expert Predictions](https://mlsanity.com/paper/2304.07306/) (2023)
- [How Uniform Random Weights Induce Non-uniform Bias: Typical Interpolating Neural Networks Generalize with Narrow Teachers](https://mlsanity.com/paper/2402.06323/) (2024)
- [Bad Universal Priors and Notions of Optimality](https://mlsanity.com/paper/1510.04931/) (2015)
- [Combining Pessimism with Optimism for Robust and Efficient Model-Based Deep Reinforcement Learning](https://mlsanity.com/paper/2103.10369/) (2021)
- [Pessimism About Unknown Unknowns Inspires Conservatism](https://mlsanity.com/paper/2006.08753/) (2020)
- [Learning-to-defer for sequential medical decision-making under uncertainty](https://mlsanity.com/paper/2109.06312/) (2021)
- [Human Control: Definitions and Algorithms](https://mlsanity.com/paper/2305.19861/) (2023)
- [A Strongly Asymptotically Optimal Agent in General Environments](https://mlsanity.com/paper/1903.01021/) (2019)
- [Embedded Universal Predictive Intelligence: a coherent framework for multi-agent learning](https://mlsanity.com/paper/2511.22226/) (2025)
- [Curiosity Killed or Incapacitated the Cat and the Asymptotically Optimal Agent](https://mlsanity.com/paper/2006.03357/) (2020)
- [Safe Exploration via Policy Priors](https://mlsanity.com/paper/2601.19612/) (2026)
- [Learning a Pessimistic Reward Model in RLHF](https://mlsanity.com/paper/2505.20556/) (2025)
- [Universal Reinforcement Learning Algorithms: Survey and Experiments](https://mlsanity.com/paper/1705.10557/) (2017)
- [Fully General Online Imitation Learning](https://mlsanity.com/paper/2102.08686/) (2021)
- [Learning to Partially Defer for Sequences](https://mlsanity.com/paper/2502.01459/) (2025)
- [Value Under Ignorance in Universal Artificial Intelligence](https://mlsanity.com/paper/2512.17086/) (2025)
- [RL, but don't do anything I wouldn't do](https://mlsanity.com/paper/2410.06213/) (2024)
- [Universal probability-free prediction](https://mlsanity.com/paper/1603.04283/) (2016)

## Built on by (most cited first, 1 of 1)

- [Understanding Generalization Requires Universal Induction](https://mlsanity.com/paper/2609.34458/) (2026)

## Abstract (arXiv)

Reinforcement learners can attain high reward through novel unintended strategies. We study a Bayesian mitigation for general environments: we expand the agent's subjective reward range to include a large negative value -L, while the true environment's rewards lie in [0,1]. After observing consistently high rewards, the Bayesian policy becomes risk-averse to novel schemes that plausibly lead to -L. We design a simple override mechanism that yields control to a safe mentor whenever the predicted value drops below a fixed threshold. We prove two properties of the resulting agent: (i) Capability: using mentor-guided exploration with vanishing frequency, the agent attains sublinear regret against its best mentor. (ii) Safety: no decidable low-complexity predicate is triggered by the optimizing policy before it is triggered by a mentor.