# Multi-Agent Security Tax: Trading Off Security and Collaboration Capabilities in Multi-Agent Systems

arXiv: https://arxiv.org/abs/2502.19145 · HTML: https://mlsanity.com/paper/2502.19145/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 40 references that are in the map.
- First posted on arXiv 26 Feb 2025.

## Builds on (most cited first, 20 of 20)

- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023)
- [HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face](https://mlsanity.com/paper/2303.17580/) (2023)
- [ExpeL: LLM Agents Are Experiential Learners](https://mlsanity.com/paper/2308.10144/) (2023)
- [Reasoning with Language Model is Planning with World Model](https://mlsanity.com/paper/2305.14992/) (2023)
- [Prompt Injection attack against LLM-integrated Applications](https://mlsanity.com/paper/2306.05499/) (2023)
- [Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaboration](https://mlsanity.com/paper/2307.05300/) (2023)
- [A Survey on Retrieval-Augmented Text Generation](https://mlsanity.com/paper/2202.01110/) (2022)
- [Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agents](https://mlsanity.com/paper/2402.11208/) (2024)
- [Evil Geniuses: Delving into the Safety of LLM-based Agents](https://mlsanity.com/paper/2311.11855/) (2023)
- [Refusal-Trained LLMs Are Easily Jailbroken As Browser Agents](https://mlsanity.com/paper/2410.13886/) (2024)
- [TrustAgent: Towards Safe and Trustworthy LLM-based Agents](https://mlsanity.com/paper/2402.01586/) (2024)
- [Here Comes The AI Worm: Unleashing Zero-click Worms that Target GenAI-Powered Applications](https://mlsanity.com/paper/2403.02817/) (2024)
- [Self-Adaptive Large Language Model (LLM)-Based Multiagent Systems](https://mlsanity.com/paper/2307.06187/) (2023)
- [Robust Safety Classifier for Large Language Models: Adversarial Prompt Shield](https://mlsanity.com/paper/2311.00172/) (2023)
- [Applying Refusal-Vector Ablation to Llama 3.1 70B Agents](https://mlsanity.com/paper/2410.10871/) (2024)

## Built on by (most cited first, 27 of 27)

- [Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents](https://mlsanity.com/paper/2505.02077/) (2025)
- [Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges](https://mlsanity.com/paper/2510.23883/) (2025)
- [A Survey of Agentic AI and Cybersecurity: Challenges, Opportunities and Use-case Prototypes](https://mlsanity.com/paper/2601.05293/) (2026)
- [From Glue-Code to Protocols: A Critical Analysis of A2A and MCP Integration for Scalable Agent Systems](https://mlsanity.com/paper/2505.03864/) (2025)
- [Inherent and emergent liability issues in LLM-based agentic systems: a principal-agent perspective](https://mlsanity.com/paper/2504.03255/) (2025)
- [Seven Security Challenges in Cross-domain Multi-agent LLM Systems](https://mlsanity.com/paper/2505.23847/) (2025)
- [The Trust Paradox in LLM-Based Multi-Agent Systems: When Collaboration Becomes a Security Vulnerability](https://mlsanity.com/paper/2510.18563/) (2025)
- [SoK: Trust-Authorization Mismatch in LLM Agent Interactions](https://mlsanity.com/paper/2512.06914/) (2025)
- [Web Fraud Attacks Against LLM-Driven Multi-Agent Systems](https://mlsanity.com/paper/2509.01211/) (2025)
- [LLM Applications: Current Paradigms and the Next Frontier](https://mlsanity.com/paper/2503.04596/) (2025)
- [Exposing Weak Links in Multi-Agent Systems under Adversarial Prompting](https://mlsanity.com/paper/2511.10949/) (2025)
- [MASTER: Multi-Agent Security Through Exploration of Roles and Topological Structures -- A Comprehensive Framework](https://mlsanity.com/paper/2505.18572/) (2025)
- [Towards Security-Auditable LLM Agents: A Unified Graph Representation](https://mlsanity.com/paper/2605.06812/) (2026)
- [Do Coding Agents Understand Least-Privilege Authorization?](https://mlsanity.com/paper/2605.14859/) (2026)
- [Formalizing the Safety, Security, and Functional Properties of Agentic AI Systems](https://mlsanity.com/paper/2510.14133/) (2025)
- [LumiMAS: A Comprehensive Framework for Real-Time Monitoring and Enhanced Observability in Multi-Agent Systems](https://mlsanity.com/paper/2508.12412/) (2025)
- [Mind Viruses: Self-Propagating Ideas in Multi-Agent LLM Systems](https://mlsanity.com/paper/2608.10218/) (2026)
- [Architecture Matters for Multi-Agent Security](https://mlsanity.com/paper/2604.23459/) (2026)
- [Cowpox: Towards the Immunity of VLM-based Multi-Agent Systems](https://mlsanity.com/paper/2508.09230/) (2025)
- [From Secure Agentic AI to Secure Agentic Web: Challenges, Threats, and Future Directions](https://mlsanity.com/paper/2603.01564/) (2026)
- [SoK: When Safe Agents Fail Together: The Security of Multi Agent LLM Systems](https://mlsanity.com/paper/2609.00595/) (2026)
- [Misleading the Planner through Deceptive Resumes: Registration-Time Injection in Centralized Multi-Agent Systems](https://mlsanity.com/paper/2609.15516/) (2026)
- [The Wisdom of Agent Crowds: A Human-AI Interaction Innovation Ignition Framework](https://mlsanity.com/paper/2505.06947/) (2025)
- [Don't Guess, Escalate: Towards Explainable Uncertainty-Calibrated AI Forensic Agents](https://mlsanity.com/paper/2512.16614/) (2025)
- [Securing Computer-Use Agents: A Unified Architecture-Lifecycle Framework for Deployment-Grounded Reliability](https://mlsanity.com/paper/2605.07110/) (2026)
- [On Understanding, Identifying, and Mitigating Vulnerabilities in Agentic Large Language Models](https://mlsanity.com/paper/2608.10530/) (2026)
- [Collective Loss of Control in LLM Agent Systems: An Epidemic Account of Mutation, Contagion, and Recovery](https://mlsanity.com/paper/2609.18460/) (2026)

## Abstract (arXiv)

As AI agents are increasingly adopted to collaborate on complex objectives, ensuring the security of autonomous multi-agent systems becomes crucial. We develop simulations of agents collaborating on shared objectives to study these security risks and security trade-offs. We focus on scenarios where an attacker compromises one agent, using it to steer the entire system toward misaligned outcomes by corrupting other agents. In this context, we observe infectious malicious prompts - the multi-hop spreading of malicious instructions. To mitigate this risk, we evaluated several strategies: two "vaccination" approaches that insert false memories of safely handling malicious input into the agents' memory stream, and two versions of a generic safety instruction strategy. While these defenses reduce the spread and fulfillment of malicious instructions in our experiments, they tend to decrease collaboration capability in the agent network. Our findings illustrate potential trade-off between security and collaborative efficiency in multi-agent systems, providing insights for designing more secure yet effective AI collaborations.