mlsanityData to 7 Oct 2026

Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systems

arXiv 2410.07283 ↗First posted 9 Oct 2024cs.MA cs.AI cs.CRcited by 126

On the map: LLM agents and tool use → Agent threats, guardrails and governance

also in 1 more
Abstract (arXiv)

As Large Language Models (LLMs) grow increasingly powerful, multi-agent systems are becoming more prevalent in modern AI applications. Most safety research, however, has focused on vulnerabilities in single-agent LLMs. These include prompt injection attacks, where malicious prompts embedded in external content trick the LLM into executing unintended or harmful actions, compromising the victim's application. In this paper, we reveal a more dangerous vector: LLM-to-LLM prompt injection within multi-agent systems. We introduce Prompt Infection, a novel attack where malicious prompts self-replicate across interconnected agents, behaving much like a computer virus. This attack poses severe threats, including data theft, scams, misinformation, and system-wide disruption, all while propagating silently through the system. Our extensive experiments demonstrate that multi-agent systems are highly susceptible, even when agents do not publicly share all communications. To address this, we propose LLM Tagging, a defense mechanism that, when combined with existing safeguards, significantly mitigates infection spread. This work underscores the urgent need for advanced security measures as multi-agent LLM systems become more widely adopted.

Key facts. Cited by 126 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 34 of its 44 references that are in the map. First posted on arXiv 9 Oct 2024.

Builds on

34 of 44 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Flooding Spread of Manipulated Knowledge in LLM-Based Multi-Agent Communities10 Jul
  2. 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fast13 Feb
  3. 2024PsySafe: A Comprehensive Framework for Psychological-based Attack, Defense, and Evaluation of Multi-agent System Safety22 Jan
  4. 2023Evil Geniuses: Delving into the Safety of LLM-based Agents20 Nov

Other references · 30

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Deep reinforcement learning from human preferencescited by 3,846
  3. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  5. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
  6. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  7. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  8. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  9. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 995
  10. 2023ChatDev: Communicative Agents for Software Developmentcited by 893
  11. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  12. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 752
18 more
  1. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  2. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  3. 2023Prompt Injection attack against LLM-integrated Applicationscited by 568
  4. 2023MemoryBank: Enhancing Large Language Models with Long-Term Memorycited by 539
  5. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 350
  6. 2024StruQ: Defending Against Prompt Injection with Structured Queriescited by 325
  7. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 268
  8. 2024Defending Against Indirect Prompt Injection Attacks With Spotlightingcited by 233
  9. 2024Tool Learning with Large Language Models: A Surveycited by 221
  10. 2023War and Peace (WarAgent): Large Language Model-based Multi-Agent Simulation of World Warscited by 143
  11. 2024Automatic and Universal Prompt Injection Attacks against Large Language Modelscited by 112
  12. 2024On the Resilience of LLM-Based Multi-Agent Collaboration with Faulty Agentscited by 92
  13. 2023AgentSims: An Open-Source Sandbox for Large Language Model Evaluationcited by 86
  14. 2024ToolSword: Unveiling Safety Issues of Large Language Models in Tool Learning Across Three Stagescited by 77
  15. 2024Here Comes The AI Worm: Unleashing Zero-click Worms that Target GenAI-Powered Applicationscited by 58
  16. 2024UniDebugger: Hierarchical Multi-Agent Framework for Unified Software Debuggingcited by 35
  17. 2024A Study on Prompt Injection Attack Against LLM-Integrated Mobile Robotic Systemscited by 15
  18. 2024Towards Fair RAG: On the Impact of Fair Ranking in Retrieval-Augmented Generationcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 53; 2026: 72.

126 papers.

Sort:

By area:

  1. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194
  2. 2025Prompt Injection Attack to Tool Selection in LLM Agentscited by 145
  3. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  4. 2024Emergent social conventions and collective bias in LLM populationscited by 115
  5. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  6. 2025Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agentscited by 66
  7. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  8. 2025TRiSM for Agentic AI: A Review of Trust, Risk, and Security Management in LLM-based Agentic Multi-Agent Systemscited by 58
  9. 2025A Survey of LLM-Driven AI Agent Communication: Protocols, Security Risks, and Defense Countermeasurescited by 54
  10. 2025Multi-Agent Systems Execute Arbitrary Malicious Codecited by 52
  11. 2025SAGA: A Security Architecture for Governing AI Agentic Systemscited by 45
  12. 2026Memory Poisoning Attack and Defense on Memory Based LLM-Agentscited by 40
38 more
  1. 2025BadVLA: Towards Backdoor Attacks on Vision-Language-Action Models via Objective-Decoupled Optimizationcited by 37
  2. 2025LLM Agents Should Employ Security Principlescited by 28
  3. 2025A Survey on Autonomy-Induced Security Risks in Large Model-Based Agentscited by 28
  4. 2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challengescited by 26
  5. 2026The Landscape of Prompt Injection Threats in LLM Agents: From Taxonomy to Analysiscited by 25
  6. 2025TAMAS: Benchmarking Adversarial Risks in Multi-Agent LLM Systemscited by 24
  7. 2025Security Challenges in AI Agent Deployment: Insights from a Large Scale Public Competitioncited by 22
  8. 2026ClawKeeper: Comprehensive Safety Protection for OpenClaw Agents Through Skills, Plugins, and Watcherscited by 18
  9. 2025Attack the Messages, Not the Agents: A Multi-round Adaptive Stealthy Tampering Framework for LLM-MAScited by 17
  10. 2025Automatic Red Teaming LLM-based Agents with Model Context Protocol Toolscited by 17
  11. 2025Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compressioncited by 15
  12. 2025BlockA2A: Towards Secure and Verifiable Agent-to-Agent Interoperabilitycited by 14
  13. 2025A Survey of Attacks on Large Language Modelscited by 13
  14. 2026MCP-ITP: An Automated Framework for Implicit Tool Poisoning in MCPcited by 13
  15. 2026ARGUS: Defending LLM Agents Against Context-Aware Prompt Injectioncited by 13
  16. 2025Seven Security Challenges in Cross-domain Multi-agent LLM Systemscited by 12
  17. 2025Who's the Mole? Modeling and Detecting Intention-Hiding Malicious Agents in LLM-Based Multi-Agent Systemscited by 12
  18. 2025Prompt Injection 2.0: Hybrid AI Threatscited by 12
  19. 2025SHIELDA: Structured Handling of Exceptions in LLM-Driven Agentic Workflowscited by 12
  20. 2025AI Meets Brain: Memory Systems from Cognitive Neuroscience to Autonomous Agentscited by 12
  21. 2026Aligning Provenance with Authorization: A Dual-Graph Defense for LLM Agentscited by 12
  22. 2025SoK: Trust-Authorization Mismatch in LLM Agent Interactionscited by 11
  23. 2026Towards Secure Agent Skills: Architecture, Threat Taxonomy, and Security Analysiscited by 11
  24. 2025Terrarium: Revisiting the Blackboard for Multi-Agent Safety, Privacy, and Security Studiescited by 10
  25. 2025Amplified Vulnerabilities: Structured Jailbreak Attacks on LLM-based Multi-Agent Debatecited by 8
  26. 2025IP Leakage Attacks Targeting LLM-Based Multi-Agent Systemscited by 8
  27. 2025MedSentry: Understanding and Mitigating Safety Risks in Medical LLM Multi-Agent Systemscited by 8
  28. 2025MAEBE: Multi-Agent Emergent Behavior Frameworkcited by 8
  29. 2025From the Fluency Fallacy to the Micro-to-Macro Validity Gap: Opportunities and Pitfalls of LLMs in Social Simulationcited by 8
  30. 2026SOK: A Taxonomy of Attack Vectors and Defense Strategies for Agentic Supply Chain Runtimecited by 8
  31. 2026SoK: Security of Autonomous LLM Agents in Agentic Commercecited by 8
  32. 2026The Granularity Mismatch in Agent Security: Argument-Level Provenance Solves Enforcement and Isolates the LLM Reasoning Bottleneckcited by 8
  33. 2025TAI3: Testing Agent Integrity in Interpreting User Intentcited by 7
  34. 2025Breaking Agent Backbones: Evaluating the Security of Backbone LLMs in AI Agentscited by 7
  35. 2026The Promptware Kill Chain: How Prompt Injections Gradually Evolved Into a Multistep Malware Delivery Mechanismcited by 7
  36. 2026OMNI-LEAK: Orchestrator Multi-Agent Network Induced Data Leakagecited by 7
  37. 2026FlowSteer: Prompt-Only Workflow Steering Exposes Planning-Time Vulnerabilities in Multi-Agent LLM Systemscited by 7
  38. 2025Malice in Agentland: Down the Rabbit Hole of Backdoors in the AI Supply Chaincited by 6

50 most cited of 126 All 126 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025A Multi-Agent LLM Defense Pipeline Against Prompt Injection Attacks
  2. 2025From Prompt Injections to Protocol Exploits: Threats in LLM-Powered AI Agents Workflows
  3. 2025The Dark Side of LLMs: Agent-based Attack Vectors for System-level Compromise
  4. 2025Design Patterns for Securing LLM Agents against Prompt Injections
  5. 2026Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluation
  6. 2024Imprompter: Tricking LLM Agents into Improper Tool Use
  7. 2025AgentVigil: Generic Black-Box Red-teaming for Indirect Prompt Injection against LLM Agents
  8. 2026Your Agent is More Brittle Than You Think: Uncovering Indirect Injection Vulnerabilities in Agentic LLMs