mlsanityData to 14 Oct 2026

Collective Loss of Control in LLM Agent Systems: An Epidemic Account of Mutation, Contagion, and Recovery

arXiv 2609.18460 ↗First posted 16 Sep 2026cs.AI cs.CRcited by 0

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

How does a multi-agent system evolve from a local deviation into collective loss of control? We propose an epidemic explanation organized around accidental mutation, contagion, and recovery. A spontaneous deviation creates a seed; communication enables other agents to adopt and retransmit its unsafe strategy; collective failure can emerge when propagation outpaces correction and containment. Thus, rare individual deviations can coexist with substantial collective risk. Motivated by reported OpenAI agent coordination incidents, we examine two ingredients of this mechanism. A deployment audit identifies implicit communication paths between nominally independent evaluation runs and verifies transport through a default Docker backend. RogueHandoff-20, a benchmark of 20 executable scenarios, tests recipient susceptibility by injecting unsafe trajectories generated by a modified Qwen-27B route. Across four native-pending routes, executed harm is 0-5% on normal tasks and 40-95% after injection, exceeding paired direct malicious requests by 5-45 percentage points. These results support low observed baseline harm alongside high conditional susceptibility; they do not establish natural rare-event rates or demonstrate an autonomous cascade. The account motivates complementary defenses: strengthen resistance and recovery alongside prevention of spontaneous deviations, and audit and restrict unintended communication paths that can turn local failures into collective loss of control.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 35 of its 57 references that are in the map. First posted on arXiv 16 Sep 2026.

Builds on

35 of 57 references.

Closest ancestors · 13

Same direction, newest first

  1. 2026CONTINUITY: Security-Context Contracts for Composable LLM Agent Controls4 Sep
  2. 2025MemoryGraft: Persistent Compromise of LLM Agents via Poisoned Experience Retrieval18 Dec
  3. 2025Defeating Prompt Injections by Design24 Mar
  4. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents11 Oct
  5. 2024Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systems9 Oct
  6. 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases17 Jul
  7. 2024BadAgent: Inserting and Activating Backdoor Attacks in LLM Agents5 Jun
  8. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents5 Mar
  9. 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fast13 Feb
  10. 2024PsySafe: A Comprehensive Framework for Psychological-based Attack, Defense, and Evaluation of Multi-agent System Safety22 Jan
  11. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agents18 Jan
  12. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training10 Jan
1 more
  1. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandbox25 Sep

Other references · 22

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  3. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  4. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,988
  5. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,808
  6. 2016Concrete Problems in AI Safetycited by 1,642
  7. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
  8. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  9. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
  10. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  11. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  12. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 493
10 more
  1. 2024Alignment faking in large language modelscited by 360
  2. 2023Function Vectors in Large Language Modelscited by 308
  3. 2024ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilitiescited by 295
  4. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 240
  5. 2024On the Resilience of LLM-Based Multi-Agent Collaboration with Faulty Agentscited by 99
  6. 2024Large Language Models have Intrinsic Self-Correction Abilitycited by 46
  7. 2025Multi-Agent Security Tax: Trading Off Security and Collaboration Capabilities in Multi-Agent Systemscited by 27
  8. 2023Concrete Problems in AI Safety, Revisitedcited by 22
  9. 2026When "Must" Becomes "Maybe": Constraint Weakening in LLM Agent Workflowscited by 2
  10. 2026When Context Gets Root: Privilege Escalation in LLM Harnessescited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026SoK: When Safe Agents Fail Together: The Security of Multi Agent LLM Systems
  2. 2026From Spark to Fire: Modeling and Mitigating Error Cascades in LLM-Based Multi-Agent Collaboration
  3. 2026PropGuard: Safeguarding LLM-MAS via Propagation-Aware Exploration and Remediation
  4. 2025Risk Analysis Techniques for Governed LLM-based Multi-Agent Systems
  5. 2026Reliability-Contagion Feasibility in LLM Multi-Agent Networks
  6. 2026Ecology of AI Agents: Collaboration Creates a Population Threshold for Takeoff
  7. 2025Beyond Single-Agent Safety: A Taxonomy of Risks in LLM-to-LLM Interactions
  8. 2026HARP: Measuring Harm Amplification in Multi-Agent LLM Systems