Password-Activated Shutdown Protocols for Misaligned Frontier Agents
On the map: LLM safety and security → Prompt injection and agent security
Abstract (arXiv)
Frontier AI developers may fail to align or control highly-capable AI agents. In many cases, it could be useful to have emergency shutdown mechanisms which effectively prevent misaligned agents from carrying out harmful actions in the world. We introduce password-activated shutdown protocols (PAS protocols) -- methods for designing frontier agents to implement a safe shutdown protocol when given a password. We motivate PAS protocols by describing intuitive use-cases in which they mitigate risks from misaligned systems that subvert other control efforts, for instance, by disabling automated monitors or self-exfiltrating to external data centres. PAS protocols supplement other safety efforts, such as alignment fine-tuning or monitoring, contributing to defence-in-depth against AI risk. We provide a concrete demonstration in SHADE-Arena, a benchmark for AI monitoring and subversion capabilities, in which PAS protocols supplement monitoring to increase safety with little cost to performance. Next, PAS protocols should be robust to malicious actors who want to bypass shutdown. Therefore, we conduct a red-team blue-team game between the developers (blue-team), who must implement a robust PAS protocol, and a red-team trying to subvert the protocol. We conduct experiments in a code-generation setting, finding that there are effective strategies for the red-team, such as using another model to filter inputs, or fine-tuning the model to prevent shutdown behaviour. We then outline key challenges to implementing PAS protocols in real-life systems, including: security considerations of the password and decisions regarding when, and in which systems, to use them. PAS protocols are an intuitive mechanism for increasing the safety of frontier AI. We encourage developers to consider implementing PAS protocols prior to internal deployment of particularly dangerous systems to reduce loss-of-control risks.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 45 of its 66 references that are in the map. First posted on arXiv 29 Nov 2025.
Builds on
45 of 66 references.
Closest ancestors · 3
Same direction, newest first
- 2025A sketch of an AI control safety case28 Jan
- 2024Games for AI Control: Models of Safety Evaluations of AI Deployment Protocols12 Sep
- 2023AI Control: Improving Safety Despite Intentional Subversion12 Dec
Other references · 42
- 2017Deep reinforcement learning from human preferencescited by 3,846
- 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
- 2021Program Synthesis with Large Language Modelscited by 3,509
- 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
- 2016Cooperative Inverse Reinforcement Learningcited by 298
- 2024Deliberative Alignment: Reasoning Enables Safer Language Modelscited by 292
- 2023Provable Robust Watermarking for AI-Generated Textcited by 259
- 2023Poisoning Language Models During Instruction Tuningcited by 254
- 2024Frontier Models are Capable of In-context Schemingcited by 252
- 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 250
- 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
- 2023An Overview of Catastrophic AI Riskscited by 214
30 more
- 2025Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teamingcited by 172
- 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
- 2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 158
- 2023Harms from Increasingly Agentic Algorithmic Systemscited by 134
- 2023Taken out of context: On measuring situational awareness in LLMscited by 131
- 2019Optimal Policies Tend to Seek Powercited by 115
- 2024Looking Inward: Language Models Can Learn About Themselves by Introspectioncited by 105
- 2022Is Power-Seeking AI an Existential Risk?cited by 101
- 2016The Off-Switch Gamecited by 97
- 2023Evaluating Language-Model Agents on Realistic Autonomous Taskscited by 94
- 2023Backdoor Attacks for In-Context Learning with Language Modelscited by 92
- 2023Scheming AIs: Will AIs fake alignment during training in order to get power?cited by 90
- 2025Tell me about yourself: LLMs are aware of their learned behaviorscited by 86
- 2024MarkLLM: An Open-Source Toolkit for LLM Watermarkingcited by 80
- 2025Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path?cited by 70
- 2024Stress-Testing Capability Elicitation With Password-Locked Modelscited by 42
- 2023A Comprehensive Overview of Backdoor Attacks in Large Language Models within Communication Networkscited by 38
- 2024A Survey of Recent Backdoor Attacks and Defenses in Large Language Modelscited by 36
- 2024Uncovering Deceptive Tendencies in Language Models: A Simulated Company AI Assistantcited by 32
- 2025Evaluating Frontier Models for Stealth and Situational Awarenesscited by 30
- 2023Analyzing And Editing Inner Mechanisms Of Backdoored Language Modelscited by 22
- 2017Should Robots be Obedient?cited by 20
- 2024Trojan Detection in Large Language Models: Insights from The Trojan Detection Challengecited by 14
- 2024Future Events as Backdoor Triggers: Investigating Temporal Vulnerabilities in LLMscited by 13
- 2023Human Control: Definitions and Algorithmscited by 10
- 2017Incorrigibility in the CIRL Frameworkcited by 9
- 2025Mechanisms to Verify International Agreements About AI Developmentcited by 9
- 2025AI Behind Closed Doors: a Primer on The Governance of Internal Deploymentcited by 8
- 2019Corrigibility with Utility Preservationcited by 7
- 2024Unelicitable Backdoors in Language Models via Cryptographic Transformer Circuitscited by 3
Built on by
Not cited yet.