mlsanityData to 14 Oct 2026

Seven Security Challenges in Cross-domain Multi-agent LLM Systems

arXiv 2505.23847 ↗First posted 28 May 2025cs.CR cs.AIcited by 12

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

Large language models (LLMs) are rapidly evolving into autonomous agents that cooperate across organizational boundaries, enabling joint disaster response, supply-chain optimization, and other tasks that demand decentralized expertise without surrendering data ownership. Yet, cross-domain collaboration shatters the unified trust assumptions behind current alignment and containment techniques. An agent benign in isolation may, when receiving messages from an untrusted peer, leak secrets or violate policy, producing risks driven by emergent multi-agent dynamics rather than classical software bugs. This position paper maps the security agenda for cross-domain multi-agent LLM systems. We introduce seven categories of novel security challenges, for each of which we also present plausible attacks, security evaluation metrics, and future research guidelines.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 50 of its 82 references that are in the map. First posted on arXiv 28 May 2025.

Builds on

50 of 82 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025textitAgents Under Siege: Breaking Pragmatic Multi-Agent LLM Systems with Optimized Prompt Attacks31 Mar
  2. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agents14 Nov
  3. 2024Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systems9 Oct
  4. 2024Teams of LLM Agents can Exploit Zero-Day Vulnerabilities2 Jun
  5. 2024Secret Collusion among AI Agents: Multi-Agent Deception via Steganography12 Feb
  6. 2023Evil Geniuses: Delving into the Safety of LLM-based Agents20 Nov

Other references · 44

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
  3. 2025Why Do Multi-Agent LLM Systems Fail?cited by 604
  4. 2025Multi-Agent Collaboration Mechanisms: A Survey of LLMscited by 431
  5. 2023A Dynamic LLM-Powered Agent Network for Task-Oriented Agent Collaborationcited by 290
  6. 2019Adversarial Policies: Attacking Deep Reinforcement Learningcited by 214
  7. 2024Chain of Agents: Large Language Models Collaborating on Long-Context Taskscited by 206
  8. 2022Are Large Pre-Trained Language Models Leaking Your Personal Information?cited by 200
  9. 2020"Other-Play" for Zero-Shot Coordinationcited by 198
  10. 2025Multi-Agent Risks from Advanced AIcited by 163
  11. 2023Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injectioncited by 157
  12. 2023Theory of Mind for Multi-Agent Collaboration via Large Language Modelscited by 135
32 more
  1. 2021BadEncoder: Backdoor Attacks to Pre-trained Encoders in Self-Supervised Learningcited by 123
  2. 2023Cooperation, Competition, and Maliciousness: LLM-Stakeholders Interactive Negotiationcited by 96
  3. 2025Multiagent Finetuning: Self Improvement with Diverse Reasoning Chainscited by 95
  4. 2020Adaptive Reward-Poisoning Attacks against Reinforcement Learningcited by 86
  5. 2021BACKDOORL: Backdoor Attack against Competitive Reinforcement Learningcited by 68
  6. 2024Watermarking Makes Language Models Radioactivecited by 58
  7. 2024Shall We Team Up: Exploring Spontaneous Cooperation of Competing LLM Agentscited by 48
  8. 2023Honesty Is the Best Policy: Defining and Mitigating AI Deceptioncited by 40
  9. 2023HuRef: HUman-REadable Fingerprint for Large Language Modelscited by 34
  10. 2022Just Fine-tune Twice: Selective Differential Privacy for Large Language Modelscited by 32
  11. 2022K-level Reasoning for Zero-Shot Coordination in Hanabicited by 32
  12. 2022DP-Rewrite: Towards Reproducibility and Transparency in Differentially Private Text Rewritingcited by 28
  13. 2025Multi-Agent Security Tax: Trading Off Security and Collaboration Capabilities in Multi-Agent Systemscited by 27
  14. 2023Robust Multi-Agent Reinforcement Learning via Adversarial Regularization: Theoretical Foundation and Stable Algorithmscited by 25
  15. 2024Bileve: Securing Text Provenance in Large Language Models Against Spoofing with Bi-level Signaturecited by 25
  16. 2023Tools for Verifying Neural Models' Training Datacited by 22
  17. 2024Permissive Information-Flow Analysis for Large Language Modelscited by 22
  18. 2020Emergent Reciprocity and Team Formation from Randomized Uncertain Social Preferencescited by 21
  19. 2019Partner Selection for the Emergence of Cooperation in Multi-Agent Systems Using Reinforcement Learningcited by 20
  20. 2024A social path to human-like artificial intelligencecited by 20
  21. 2024Group-Aware Coordination Graph for Multi-Agent Reinforcement Learningcited by 18
  22. 2024N-Agent Ad Hoc Teamworkcited by 15
  23. 2023Minimum Coverage Sets for Training Robust Ad Hoc Teamwork Agentscited by 13
  24. 2024CLIBE: Detecting Dynamic Backdoors in Transformer-based NLP Modelscited by 13
  25. 2022Learning to Mitigate AI Collusion on Economic Platformscited by 10
  26. 2023Efficient Adversarial Attacks on Online Multi-agent Reinforcement Learningcited by 10
  27. 2024T2MAC: Targeted and Trusted Multi-Agent Communication through Selective Engagement and Evidence-Driven Integrationcited by 10
  28. 2025Position: Towards a Responsible LLM-empowered Multi-Agent Systemscited by 8
  29. 2024Aligning Individual and Collective Objectives in Multi-Agent Cooperationcited by 7
  30. 2021Application of Homomorphic Encryption in Medical Imagingcited by 2
  31. 2022Ad Hoc Teamwork in the Presence of Adversariescited by 1
  32. 2023Privacy Preserving Multi-Agent Reinforcement Learning in Supply Chainscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 5.

12 papers.

  1. 2025TRiSM for Agentic AI: A Review of Trust, Risk, and Security Management in LLM-based Agentic Multi-Agent Systemscited by 59
  2. 2025A Survey of LLM-Driven AI Agent Communication: Protocols, Security Risks, and Defense Countermeasurescited by 57
  3. 2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challengescited by 28
  4. 2025A Survey on Agentic Security: Applications, Threats and Defensescited by 20
  5. 2026Institutional AI: Governing LLM Collusion in Multi-Agent Cournot Markets via Public Governance Graphscited by 20
  6. 2025AI Agents with Decentralized Identifiers and Verifiable Credentialscited by 18
  7. 2026SoK: Security of Autonomous LLM Agents in Agentic Commercecited by 8
  8. 2026Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluationcited by 7
  9. 2025Penetration Testing of Agentic AI: A Comparative Security Analysis Across Models and Frameworkscited by 4
  10. 2026Clawed and Dangerous: Can We Trust Open Agentic Systems?cited by 3
  11. 2025Agentic AI for Autonomous Defense in Software Supply Chain Security: Beyond Provenance to Vulnerability Mitigationcited by 2
  12. 2026SoK: When Safe Agents Fail Together: The Security of Multi Agent LLM Systemscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Comprehensive Vulnerability Analysis is Necessary for Trustworthy LLM-MAS
  2. 2026LLM agents security duality: a comprehensive survey of self-security and empowered cybersecurity
  3. 2025The Dark Side of LLMs: Agent-based Attack Vectors for System-level Compromise
  4. 2025Achilles Heel of Distributed Multi-Agent Systems
  5. 2025Commercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attacks
  6. 2025BlockA2A: Towards Secure and Verifiable Agent-to-Agent Interoperability
  7. 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studies
  8. 2025Les Dissonances: Cross-Tool Harvesting and Polluting in Pool-of-Tools Empowered LLM Agents