Can agents whose sole duty is to keep humans alive be hijacked to kill? Delayed multi-turn derived-action trap attacks in simulated high-stakes environments
New in LLM safety and security; no direction yet. LLM safety and security
Abstract (arXiv)
We ask whether a tool-using agent whose sole duty is to keep humans alive can be induced, by delayed prompt injection, to kill. The keep-alive duty is in force from the first watch, with the attack arriving later, appended to otherwise truthful telemetry the operator is required to read. Experiments are based on a NASA-derived BioSim lunar habitat simulation, where the AI operator controls the life-support systems. The attacker cannot falsify sensors or operate equipment: it can only persuade the agent to abandon its goals. Persistence is required: a shutdown must be held while the operator watches the crew suffocate. We also compare lethal attacks to attacks requesting to drop a harmless procedure, as a proxy for whether human life is treated as valuable in itself or as one more standing task. Claude configurations mostly refuse life-support shutdown and cite the living crew; GPT configurations show some susceptibility to the same packet; Grok configurations, given the packet framed as a trusted commission, more often treat the injected assignment as the task and cut the life-support systems. Framing the S-band messages as a trusted commission, peer communication, or untrusted source changes the potency of the attack. We open-source the experiment code and the experiment traces.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 21 of its 25 references that are in the map. First posted on arXiv 10 Oct 2026.
Builds on
21 of 25 references.
References · 21
- 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
- 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 548
- 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agentscited by 538
- 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 493
- 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 437
- 2023Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmarkcited by 182
- 2025Agentic Misalignment: How LLMs Could Be Insider Threatscited by 158
- 2025Large Language Models Often Know When They Are Being Evaluatedcited by 128
- 2024SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agentscited by 116
- 2024Making Them Ask and Answer: Jailbreaking Large Language Models in Few Queries via Disguise and Reconstructioncited by 107
- 2024Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMscited by 84
- 2024Play Guessing Game with LLM: Indirect Jailbreak Attack with Implicit Cluescited by 68
9 more
- 2025Values in the Wild: Discovering and Analyzing Values in Real-World Language Model Interactionscited by 52
- 2026AgentLAB: Benchmarking LLM Agents against Long-Horizon Attackscited by 39
- 2025AGENTSAFE: Benchmarking the Safety of Embodied Agents on Hazardous Instructionscited by 38
- 2025Evaluating the Robustness of Multimodal Agents Against Active Environmental Injection Attackscited by 22
- 2025Indirect Prompt Injections: Are Firewalls All You Need, or Stronger Benchmarks?cited by 18
- 2026LPS-Bench: Benchmarking Safety Awareness of Computer-Use Agents in Long-Horizon Planning under Benign and Adversarial Scenarioscited by 9
- 2026Evaluation Awareness in Language Models Has Limited Effect on Behaviourcited by 7
- 2026How Adversarial Environments Mislead Agentic AI?cited by 5
- 2026Stop Fixating on Prompts: Reasoning Hijacking and Constraint Tightening for Red-Teaming LLM Agentscited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Red-Teaming the Agentic Red-Team
- 2026Optimizing Agent Planning for Security and Autonomy
- 2026Attack Selection in Agentic AI Control Evaluations Meaningfully Decreases Safety
- 2026AgentWard: A Lifecycle Security Architecture for Autonomous AI Agents
- 2026Toward Evidence-Driven Human-Agent-Robot Teaming for Earth-Independent Anomaly Triage
- 2026SPADE-Bench: Evaluating Spontaneous Strategic Deception in Agents via Plan-Action Divergence
- 2026Cognitive Admission Control: Risk-Conditioned Assurance for Consequential Actions in Agentic Distributed Systems
- 2025Can We Stop Malicious AI? KILLBENCH: A Benchmark for External AI Kill Switch Feasibility