Compositional Threat Analysis of Latent Compromise in LLM Agent Systems: The Order 66 Scenario
On the map: LLM agents and tool use → Agent threats, guardrails and governance
Abstract (arXiv)
In the fictional Order 66, catastrophe does not arise from a powerful command alone: a trusted population is preconditioned, a short directive activates the concealed condition, and protective authority turns against the system. This paper translates that mechanism into an origin-neutral security analysis of tool-using large language model (LLM) agents. A representative scenario combines a deployed artifact or shared memory bearing a dormant destructive rule, a later email, document, update, or peer message that activates it, and an agent harness granting operational and recovery authority. We introduce a compositional model explaining why no component is catastrophic alone, yet their conjunction can produce correlated destructive action. We separate three population-reach routes --- release-time pre-positioning, post-release durable seeding, and peer replication --- from a common core of dormancy, activation, authority, reachable targets, and failed recovery. This yields defensive cut sets and shows why checkpoint scanning or prompt filtering cannot close every route. A two-class example shows that cross-class feedback can sustain spread even when both within-class reproduction terms are below one; isolation and persistence controls suppress the loop. Published work instantiates constituent mechanisms, while incidents demonstrate autonomous boundary crossing, malicious agent extensions, agent-assisted reconnaissance, and public-package propagation, but not the full dormant-implant composition. We found no public observation, in evidence reviewed through 5 August 2026, traversing the complete Order 66 graph. The result is neither dismissal nor prediction: the scenario is componentwise credible under stated assumptions, damage depends on the harness, and the strongest defenses are capability mediation, durable-state provenance, propagation isolation, and protected recovery.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 22 of its 59 references that are in the map. First posted on arXiv 8 Aug 2026.
Builds on
22 of 59 references.
Closest ancestors · 14
Same direction, newest first
- 2026From Untrusted Input to Trusted Memory: A Systematic Study of Memory Poisoning Attacks in LLM Agents3 Jun
- 2026AI Agents Enable Adaptive Computer Worms2 Jun
- 2026MemMorph: Tool Hijacking in LLM Agents via Memory Poisoning24 May
- 2026Hidden in Memory: Sleeper Memory Poisoning in LLM Agents14 May
- 2026Your LLM Agent Can Leak Your Data: Data Exfiltration via Backdoored Tool Use7 Apr
- 2026AgentWorm: Self-Propagating Attacks Across LLM Agent Ecosystems16 Mar
- 2026"Do Not Mention This to the User": Detecting and Understanding Malicious Agent Skills in the Wild6 Feb
- 2025MemoryGraft: Persistent Compromise of LLM Agents via Poisoned Experience Retrieval18 Dec
- 2025Progent: Securing AI Agents with Privilege Control16 Apr
- 2025Defeating Prompt Injections by Design24 Mar
- 2025Memory Injection Attacks on LLM Agents via Query-Only Interaction5 Mar
- 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases17 Jul
2 more
Other references · 8
- 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
- 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 173
- 2025The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injectionscited by 129
- 2025Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samplescited by 72
- 2024BackdoorLLM: A Comprehensive Benchmark for Backdoor Attacks and Defenses on Large Language Modelscited by 50
- 2026The Trigger in the Haystack: Extracting and Reconstructing LLM Backdoor Triggerscited by 14
- 2025TrojanStego: Your Language Model Can Secretly Be A Steganographic Privacy Leaking Agentcited by 9
- 2026The Promptware Kill Chain: How Prompt Injections Gradually Evolved Into a Multistep Malware Delivery Mechanismcited by 9
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluation
- 2026Security Engineering of OpenClaw: Analyzing Attack Surface Expansion and Trust-Boundary Violations
- 2026Safe Multi-Agent Behavior Must Be Maintained, Not Merely Asserted: Constraint Drift in LLM-Based Multi-Agent Systems
- 2025Architecting Resilient LLM Agents: A Guide to Secure Plan-then-Execute Implementations
- 2025When Refusals Fail: Unstable Safety Mechanisms in Long-Context LLM Agents
- 2026Securing LLM-Agent Long-Term Memory Against Poisoning: Non-Malleable, Origin-Bound Authority with Machine-Checked Guarantees
- 2026Mind the GAP: Text Safety Does Not Transfer to Tool-Call Safety in LLM Agents
- 2026SoK: When Safe Agents Fail Together: The Security of Multi Agent LLM Systems