mlsanityData to 14 Oct 2026

textitAgents Under Siege: Breaking Pragmatic Multi-Agent LLM Systems with Optimized Prompt Attacks

arXiv 2504.00218 ↗First posted 31 Mar 2025cs.MA cs.AI cs.CLcited by 34

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

Most discussions about Large Language Model (LLM) safety have focused on single-agent settings but multi-agent LLM systems now create novel adversarial risks because their behavior depends on communication between agents and decentralized reasoning. In this work, we innovatively focus on attacking pragmatic systems that have constrains such as limited token bandwidth, latency between message delivery, and defense mechanisms. We design a textitpermutation-invariant adversarial attack that optimizes prompt distribution across latency and bandwidth-constraint network topologies to bypass distributed safety mechanisms within the system. Formulating the attack path as a problem of textitmaximum-flow minimum-cost, coupled with the novel textitPermutation-Invariant Evasion Loss (PIEL), we leverage graph-based optimization to maximize attack success rate while minimizing detection risk. Evaluating across models including textttLlama, textttMistral, textttGemma, textttDeepSeek and other variants on various datasets like textttJailBreakBench and textttAdversarialBench, our method outperforms conventional attacks by up to 7×, exposing critical vulnerabilities in multi-agent systems. Moreover, we demonstrate that existing defenses, including variants of textttLlama-Guard and textttPromptGuard, fail to prohibit our attack, emphasizing the urgent need for multi-agent specific safety mechanisms.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 34 of its 59 references that are in the map. First posted on arXiv 31 Mar 2025.

Builds on

34 of 59 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024AutoDefense: Multi-Agent LLM Defense against Jailbreak Attacks2 Mar
  2. 2024The Wolf Within: Covert Injection of Malice into MLLM Societies via an MLLM Operative20 Feb
  3. 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fast13 Feb
  4. 2023Evil Geniuses: Delving into the Safety of LLM-based Agents20 Nov

Other references · 30

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  8. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  9. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  10. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  11. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
  12. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
18 more
  1. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  2. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 1,011
  3. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 742
  4. 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviorscited by 598
  5. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 573
  6. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 498
  7. 2023Self-collaboration Code Generation via ChatGPTcited by 258
  8. 2023Auto-GPT for Online Decision Making: Benchmarks and Additional Opinionscited by 255
  9. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 234
  10. 2023LM vs LM: Detecting Factual Errors via Cross Examinationcited by 156
  11. 2023War and Peace (WarAgent): Large Language Model-based Multi-Agent Simulation of World Warscited by 145
  12. 2023Agents: An Open-source Framework for Autonomous Language Agentscited by 116
  13. 2023Mindstorms in Natural Language-Based Societies of Mindcited by 107
  14. 2023Training Socially Aligned Language Models on Simulated Social Interactionscited by 78
  15. 2024MultiAgent Collaboration Attack: Investigating Adversarial Attacks in Large Language Model Collaborations via Debatecited by 56
  16. 2022A Study of the Attention Abnormality in Trojaned BERTscited by 48
  17. 2024Task-Agnostic Detector for Insertion-Based Backdoor Attackscited by 23
  18. 2022Poisoning Attacks and Defenses on Artificial Intelligence: A Surveycited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 24.

34 papers.

Sort:

By area:

  1. 2025A Survey of LLM-Driven AI Agent Communication: Protocols, Security Risks, and Defense Countermeasurescited by 57
  2. 2025SAGA: A Security Architecture for Governing AI Agentic Systemscited by 45
  3. 2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challengescited by 28
  4. 2025Security of Internet of Agents: Attacks and Countermeasurescited by 17
  5. 2026ClawSafety: "Safe" LLMs, Unsafe Agentscited by 17
  6. 2025BlockA2A: Towards Secure and Verifiable Agent-to-Agent Interoperabilitycited by 15
  7. 2025Seven Security Challenges in Cross-domain Multi-agent LLM Systemscited by 12
  8. 2025Amplified Vulnerabilities: Structured Jailbreak Attacks on LLM-based Multi-Agent Debatecited by 10
  9. 2026Thought Virus: Viral Misalignment via Subliminal Prompting in Multi-Agent Systemscited by 9
  10. 2025IP Leakage Attacks Targeting LLM-Based Multi-Agent Systemscited by 8
  11. 2026Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluationcited by 7
  12. 2025Agentic Services Computingcited by 6
22 more
  1. 2025Evo-MARL: Co-Evolutionary Multi-Agent Reinforcement Learning for Internalized Safetycited by 4
  2. 2026WebWeaver: Breaking Topology Confidentiality in LLM Multi-Agent Systems with Stealthy Context-Based Inferencecited by 3
  3. 2026From Logic Monopoly to Social Contract: Separation of Power and the Institutional Foundations for Autonomous Agent Economiescited by 3
  4. 2026When Embedding-Based Defenses Fail: Rethinking Safety in LLM-Based Multi-Agent Systemscited by 2
  5. 2026SeClaw: Spec-Driven Security Task Synthesis for Evaluating Autonomous Agentscited by 1
  6. 2026The Deliberative Illusion: Diagnosing Factual Attrition and Stance Homogenization in Multi-Agent LLM Deliberationcited by 1
  7. 2026When Agents Go Rogue: Activation-Based Detection of Malicious Behaviors in Multi-Agent Systemscited by 1
  8. 2026Agent Harness Distillation: Inference-Time Harness Extraction and Exploitation in Autonomous Multi-Agent Systemscited by 1
  9. 2026Adversarial Attacks in Multi-Agent LLM Pipelines: Unveiling Structural Vulnerabilities in Agentic AI Architecturescited by 1
  10. 2026When Prompts Control Robots: Prompt Injection Attacks in Multi-Agent Robotic Systemscited by 1
  11. 2026SoK: Intent-Oriented Systematization of Multi-Turn LLM Jailbreakscited by 1
  12. 2026Blockchain Empowered Trustworthy Agent Networks: Foundations, Taxonomy, and Future Directionscited by 1
  13. 2026SoK: When Safe Agents Fail Together: The Security of Multi Agent LLM Systemscited by 1
  14. 2026FLARE: Agentic Coverage-Guided Fuzzing for LLM-Based Multi-Agent Systemscited by 0
  15. 2026Cross-Agent Campaign Attribution: Linking Asynchronous Attacks Across LLM Agentscited by 0
  16. 2026Even More Deception: Objective Misalignment in Mixed-Motive LLM Multi-Agent Systemscited by 0
  17. 2026From Monoliths to Swarms: A Study of Attack Surface Evolution in the Transition to Multi-Agent Web Systemscited by 0
  18. 2026Attacking and Defending Multi-Agent Collaborative Filtering Systems Through Connectivitycited by 0
  19. 2026The Safeguard Worked. Is the LLM System Safer?cited by 0
  20. 2026Policy-Backed Selective Regeneration under Tainted Inter-Agent Communicationcited by 0
  21. 2026Concealing LLM-Based Multi-Agent Topology via Phantom Structure Injectioncited by 0
  22. 2026Forgetting is Not Always Bad: A Neuro-Inspired Memory Repair Mechanism for Poisoned LLM Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025TAMAS: Benchmarking Adversarial Risks in Multi-Agent LLM Systems
  2. 2025A Multi-Agent LLM Defense Pipeline Against Prompt Injection Attacks
  3. 2024Imprompter: Tricking LLM Agents into Improper Tool Use
  4. 2025Red-Teaming LLM Multi-Agent Systems via Communication Attacks
  5. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks
  6. 2026Conjunctive Prompt Attacks in Multi-Agent LLM Systems
  7. 2024Targeting the Core: A Simple and Effective Method to Attack RAG-based Agents via Direct LLM Manipulation
  8. 2025The Dark Side of LLMs: Agent-based Attack Vectors for System-level Compromise