mlsanityData to 14 Oct 2026

Multi-Agent Systems Execute Arbitrary Malicious Code

arXiv 2503.12188 ↗First posted 15 Mar 2025cs.CR cs.LGcited by 53

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

Multi-agent systems coordinate LLM-based agents to perform tasks on users' behalf. In real-world applications, multi-agent systems will inevitably interact with untrusted inputs, such as malicious Web content, files, email attachments, and more. Using several recently proposed multi-agent frameworks as concrete examples, we demonstrate that adversarial content can hijack control and communication within the system to invoke unsafe agents and functionalities. This results in a complete security breach, up to execution of arbitrary malicious code on the user's device or exfiltration of sensitive data from the user's containerized environment. For example, when agents are instantiated with GPT-4o, Web-based attacks successfully cause the multi-agent system execute arbitrary malicious code in 58-90% of trials (depending on the orchestrator). In some model-orchestrator configurations, the attack success rate is 100%. We also demonstrate that these attacks succeed even if individual agents are not susceptible to direct or indirect prompt injection, and even if they refuse to perform harmful actions. We hope that these results will motivate development of trust and security models for multi-agent systems before they are widely deployed.

Key facts. Cited by 53 papers in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 41 of its 82 references that are in the map. First posted on arXiv 15 Mar 2025.

Builds on

41 of 82 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024The Task Shield: Enforcing Task Alignment to Defend Against Indirect Prompt Injection in LLM Agents21 Dec
  2. 2024Imprompter: Tricking LLM Agents into Improper Tool Use19 Oct
  3. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents11 Oct
  4. 2024Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systems9 Oct
  5. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents3 Oct
  6. 2024Breaking Agents: Compromising Autonomous LLM Agents Through Malfunction Amplification30 Jul
  7. 2024Flooding Spread of Manipulated Knowledge in LLM-Based Multi-Agent Communities10 Jul
  8. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents5 Mar

Other references · 33

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  3. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  4. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,988
  5. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,808
  6. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
  7. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  8. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  9. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
  10. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
  11. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 672
  12. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 493
21 more
  1. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 486
  2. 2024StruQ: Defending Against Prompt Injection with Structured Queriescited by 375
  3. 2023MLAgentBench: Evaluating Language Agents on Machine Learning Experimentationcited by 366
  4. 2024Defending Against Indirect Prompt Injection Attacks With Spotlightingcited by 267
  5. 2024Magentic-One: A Generalist Multi-Agent System for Solving Complex Taskscited by 232
  6. 2024AI Agents That Mattercited by 219
  7. 2024SecAlign: Defending Against Prompt Injection with Preference Optimizationcited by 210
  8. 2023Harms from Increasingly Agentic Algorithmic Systemscited by 135
  9. 2024The Ethics of Advanced AI Assistantscited by 107
  10. 2023Jatmo: Prompt Injection Defense by Task-Specific Finetuningcited by 106
  11. 2024On the Resilience of LLM-Based Multi-Agent Collaboration with Faulty Agentscited by 99
  12. 2024System-Level Defense against Indirect Prompt Injection Attacks: An Information Flow Control Perspectivecited by 96
  13. 2024Neural Exec: Learning (and Learning from) Execution Triggers for Prompt Injection Attackscited by 85
  14. 2024Here Comes The AI Worm: Unleashing Zero-click Worms that Target GenAI-Powered Applicationscited by 59
  15. 2024Signed-Prompt: A New Approach to Prevent Prompt Injection Attacks Against LLM-Integrated Applicationscited by 54
  16. 2023Demystifying RCE Vulnerabilities in LLM-Integrated Appscited by 37
  17. 2024SPINACH: SPARQL-Based Information Navigation for Challenging Real-World Questionscited by 27
  18. 2024ConfusedPilot: Confused Deputy Risks in RAG-based LLMscited by 22
  19. 2025Rerouting LLM Routerscited by 11
  20. 2024Self-interpreting Adversarial Imagescited by 7
  21. 2024Privacy Bias in Language Models: A Contextual Integrity-based Auditing Metriccited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 18; 2026: 35.

53 papers.

Sort:

By area:

  1. 2025A Survey of Context Engineering for Large Language Modelscited by 126
  2. 2025A Survey of LLM-Driven AI Agent Communication: Protocols, Security Risks, and Defense Countermeasurescited by 57
  3. 2025SAGA: A Security Architecture for Governing AI Agentic Systemscited by 45
  4. 2025Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AIcited by 42
  5. 2025A Survey on Autonomy-Induced Security Risks in Large Model-Based Agentscited by 29
  6. 2025AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunitiescited by 28
  7. 2025SentinelAgent: Graph-based Anomaly Detection in Multi-Agent Systemscited by 22
  8. 2026SkillProbe: Security Auditing for Emerging Agent Skill Marketplaces via Multi-Agent Collaborationcited by 21
  9. 2025Securing GenAI Multi-Agent Systems Against Tool Squatting: A Zero Trust Registry-Based Approachcited by 18
  10. 2026Overcoming the Retrieval Barrier: Indirect Prompt Injection in the Wild for LLM Systemscited by 16
  11. 2025BlockA2A: Towards Secure and Verifiable Agent-to-Agent Interoperabilitycited by 15
  12. 2026From Spark to Fire: Modeling and Mitigating Error Cascades in LLM-Based Multi-Agent Collaborationcited by 14
38 more
  1. 2026ARGUS: Defending LLM Agents Against Context-Aware Prompt Injectioncited by 14
  2. 2025Breaking and Fixing Defenses Against Control-Flow Hijacking in Multi-Agent Systemscited by 12
  3. 2025Topology Matters: Measuring Memory Leakage in Multi-Agent LLMscited by 11
  4. 2025Terrarium: Revisiting the Blackboard for Multi-Agent Safety, Privacy, and Security Studiescited by 10
  5. 2026OMNI-LEAK: Orchestrator Multi-Agent Network Induced Data Leakagecited by 9
  6. 2025STAC: When Innocent Tools Form Dangerous Chains for LLM Agentscited by 8
  7. 2025SentinelNet: Safeguarding Multi-Agent Collaboration Through Credit-Based Dynamic Threat Detectioncited by 8
  8. 2026AgentMark: Utility-Preserving Behavioral Watermarking for Agentscited by 8
  9. 2026DiLLS: Interactive Diagnosis of LLM-based Multi-agent Systems via Layered Summary of Agent Behaviorscited by 7
  10. 2026FlowSteer: Prompt-Only Workflow Steering Exposes Planning-Time Vulnerabilities in Multi-Agent LLM Systemscited by 7
  11. 2026Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluationcited by 7
  12. 2026SafeClaw-R: Towards Safe and Secure Multi-Agent Personal Assistantscited by 6
  13. 2026Semantic Intent Fragmentation: A Single-Shot Compositional Attack on Multi-Agent AI Pipelinescited by 5
  14. 2026Sequential Behavioral Watermarking for LLM Agentscited by 5
  15. 2025Evo-MARL: Co-Evolutionary Multi-Agent Reinforcement Learning for Internalized Safetycited by 4
  16. 2026Clawed and Dangerous: Can We Trust Open Agentic Systems?cited by 3
  17. 2026Operational Reframing and Approval-Framed Delegation in Multi-Agent LLM Safetycited by 3
  18. 2025Analyzing Code Injection Attacks on LLM-based Multi-Agent Systems in Software Developmentcited by 2
  19. 2026MaMa: A Game-Theoretic Approach for Designing Safe Agentic Systemscited by 2
  20. 2026AgenticCyOps: Securing Multi-Agentic AI Integration in Enterprise Cyber Operationscited by 2
  21. 2026Conjunctive Prompt Attacks in Multi-Agent LLM Systemscited by 2
  22. 2026When Child Inherits: Modeling and Exploiting Subagent Spawn in Multi-Agent Networkscited by 2
  23. 2026Agent Meltdowns: The Road to Hell Is Paved with Helpful Agentscited by 2
  24. 2026"I Strongly Suspect This Website Is a Scam": Benchmarking PII Leakage and Detection without Defense in Autonomous Web Agentscited by 2
  25. 2025MiniScope: A Least Privilege Framework for Authorizing Tool Calling Agentscited by 1
  26. 2026Challenges and Future Directions in Agentic Reverse Engineering Systemscited by 1
  27. 2026The Containment Gap: How Deployed Agentic AI Frameworks Fail Public-Facing Safety Requirementscited by 1
  28. 2026When Prompts Control Robots: Prompt Injection Attacks in Multi-Agent Robotic Systemscited by 1
  29. 2026BRA-Audit: Budgeted Runtime Auditing for LLM Multi-Agent Systems via Cumulative-Exposure Audit-Point Placementcited by 1
  30. 2026SoK: When Safe Agents Fail Together: The Security of Multi Agent LLM Systemscited by 1
  31. 2025From Benchmark Data To Applicable Program Repair: An Experience Reportcited by 0
  32. 2026MESA: Prioritizing Vulnerable Communication Channels for Securing Multi-Agent Systemscited by 0
  33. 2026Cross-Agent Campaign Attribution: Linking Asynchronous Attacks Across LLM Agentscited by 0
  34. 2026Even More Deception: Objective Misalignment in Mixed-Motive LLM Multi-Agent Systemscited by 0
  35. 2026From Monoliths to Swarms: A Study of Attack Surface Evolution in the Transition to Multi-Agent Web Systemscited by 0
  36. 2026Attacking and Defending Multi-Agent Collaborative Filtering Systems Through Connectivitycited by 0
  37. 2026On Understanding, Identifying, and Mitigating Vulnerabilities in Agentic Large Language Modelscited by 0
  38. 2026Reflections on Trusting Trust, Revisited: Contaminating Self-Modifying AI Coding Agents with Poisoned Benchmarkscited by 0

50 most cited of 53 All 53 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Exposing Weak Links in Multi-Agent Systems under Adversarial Prompting
  2. 2025Shadows in the Code: Exploring the Risks and Defenses of LLM-based Multi-Agent Software Development Systems
  3. 2025Demonstrations of Integrity Attacks in Multi-Agent Systems
  4. 2025Red-Teaming LLM Multi-Agent Systems via Communication Attacks
  5. 2025The Dark Side of LLMs: Agent-based Attack Vectors for System-level Compromise
  6. 2025From Prompt Injections to Protocol Exploits: Threats in LLM-Powered AI Agents Workflows
  7. 2025Web Fraud Attacks Against LLM-Driven Multi-Agent Systems
  8. 2025Measuring the Security of Mobile LLM Agents under Adversarial Prompts from Untrusted Third-Party Channels