Towards Shutdownable Agents via Stochastic Choice
Not on the map. Why?
Abstract (arXiv)
The POST-Agents Proposal (PAP) is an idea for ensuring that advanced artificial agents never resist shutdown. A key part of the PAP is using a novel `Discounted Reward for Same-Length Trajectories (DReST)' reward function to train agents to (1) pursue goals effectively conditional on each trajectory-length (be `USEFUL'), and (2) choose stochastically between different trajectory-lengths (be `NEUTRAL' about trajectory-lengths). In this paper, we propose evaluation metrics for USEFULNESS and NEUTRALITY. We use a DReST reward function to train simple agents to navigate gridworlds, and we find that these agents learn to be USEFUL and NEUTRAL. Our results thus provide some initial evidence that DReST reward functions could train advanced agents to be USEFUL and NEUTRAL. Our theoretical work suggests that these agents would be useful and shutdownable.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 24 of its 80 references that are in the map. First posted on arXiv 30 Jun 2024.
Builds on
24 of 80 references.
References · 24
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2015Trust Region Policy Optimizationcited by 2,503
- 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
- 2017Constrained Policy Optimizationcited by 928
- 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
- 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
- 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
- 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 250
- 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
- 2021A Practical Guide to Multi-Objective Reinforcement Learning and Planningcited by 240
- 2021Goal Misgeneralization in Deep Reinforcement Learningcited by 122
- 2022Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goalscited by 120
12 more
- 2019Optimal Policies Tend to Seek Powercited by 115
- 2022Is Power-Seeking AI an Existential Risk?cited by 101
- 2016The Off-Switch Gamecited by 97
- 2023Managing extreme AI risks amid rapid progresscited by 48
- 2024The Shutdown Problem: An AI Engineering Puzzle for Decision Theoristscited by 22
- 2022Parametrically Retargetable Decision-Makers Tend To Seek Powercited by 19
- 2016Self-Modification of Policy and Utility Function in Rational Agentscited by 11
- 2023Human Control: Definitions and Algorithmscited by 10
- 2016Death and Suicide in Universal Artificial Intelligencecited by 9
- 2017A Game-Theoretic Analysis of the Off-Switch Gamecited by 8
- 2019Corrigibility with Utility Preservationcited by 7
- 2017'Indifference' methods for managing agent rewardscited by 4
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.