mlsanityData to 14 Oct 2026

Golden Handcuffs make safer AI agents

arXiv 2604.13609 ↗First posted 15 Apr 2026cs.LG cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Reinforcement learners can attain high reward through novel unintended strategies. We study a Bayesian mitigation for general environments: we expand the agent's subjective reward range to include a large negative value -L, while the true environment's rewards lie in [0,1]. After observing consistently high rewards, the Bayesian policy becomes risk-averse to novel schemes that plausibly lead to -L. We design a simple override mechanism that yields control to a safe mentor whenever the predicted value drops below a fixed threshold. We prove two properties of the resulting agent: (i) Capability: using mentor-guided exploration with vanishing frequency, the agent attains sublinear regret against its best mentor. (ii) Safety: no decidable low-complexity predicate is triggered by the optimizing policy before it is triggered by a mentor.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 30 of its 49 references that are in the map. First posted on arXiv 15 Apr 2026.

Builds on

30 of 49 references.

References · 30

  1. 2021Towards Out-Of-Distribution Generalization: A Surveycited by 436
  2. 2017Predict Responsibly: Improving Fairness and Accuracy by Learning to Defercited by 233
  3. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 231
  4. 2019Optimal Policies Tend to Seek Powercited by 119
  5. 2019Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspectivecited by 108
  6. 2017Trial without Error: Towards Safe Reinforcement Learning via Human Interventioncited by 103
  7. 2016The Off-Switch Gamecited by 97
  8. 2024Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systemscited by 96
  9. 2026OpenClaw-RL: Train Any Agent Simply by Talkingcited by 90
  10. 2023Non-Vacuous Generalization Bounds for Large Language Modelscited by 53
  11. 2021DOPE: Doubly Optimistic and Pessimistic Exploration for Safe Reinforcement Learningcited by 35
  12. 2023Deep neural networks have an inbuilt Occam's razorcited by 28
18 more
  1. 2023Learning to Defer with Limited Expert Predictionscited by 25
  2. 2024How Uniform Random Weights Induce Non-uniform Bias: Typical Interpolating Neural Networks Generalize with Narrow Teacherscited by 18
  3. 2015Bad Universal Priors and Notions of Optimalitycited by 16
  4. 2021Combining Pessimism with Optimism for Robust and Efficient Model-Based Deep Reinforcement Learningcited by 14
  5. 2020Pessimism About Unknown Unknowns Inspires Conservatismcited by 13
  6. 2021Learning-to-defer for sequential medical decision-making under uncertaintycited by 12
  7. 2023Human Control: Definitions and Algorithmscited by 10
  8. 2019A Strongly Asymptotically Optimal Agent in General Environmentscited by 9
  9. 2025Embedded Universal Predictive Intelligence: a coherent framework for multi-agent learningcited by 9
  10. 2020Curiosity Killed or Incapacitated the Cat and the Asymptotically Optimal Agentcited by 6
  11. 2026Safe Exploration via Policy Priorscited by 6
  12. 2025Learning a Pessimistic Reward Model in RLHFcited by 5
  13. 2017Universal Reinforcement Learning Algorithms: Survey and Experimentscited by 4
  14. 2021Fully General Online Imitation Learningcited by 4
  15. 2025Learning to Partially Defer for Sequencescited by 4
  16. 2025Value Under Ignorance in Universal Artificial Intelligencecited by 3
  17. 2024RL, but don't do anything I wouldn't docited by 2
  18. 2016Universal probability-free predictioncited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Understanding Generalization Requires Universal Inductioncited by 0