Exposing Weak Links in Multi-Agent Systems under Adversarial Prompting
On the map: LLM agents and tool use → Agent threats, guardrails and governance
also in 1 more
Abstract (arXiv)
LLM-based agents are increasingly deployed in multi-agent systems (MAS). As these systems move toward real-world applications, their security becomes paramount. Existing research largely evaluates single-agent security, leaving a critical gap in understanding the vulnerabilities introduced by multi-agent design. However, existing systems fall short due to lack of unified frameworks and metrics focusing on unique rejection modes in MAS. We present SafeAgents, a unified and extensible framework for fine-grained security assessment of MAS. SafeAgents systematically exposes how design choices such as plan construction strategies, inter-agent context sharing, and fallback behaviors affect susceptibility to adversarial prompting. We introduce Dharma, a diagnostic measure that helps identify weak links within multi-agent pipelines. Using SafeAgents, we conduct a comprehensive study across five widely adopted multi-agent architectures (centralized, decentralized, and hybrid variants) on four datasets spanning web tasks, tool use, and code generation. Our findings reveal that common design patterns carry significant vulnerabilities. For example, centralized systems that delegate only atomic instructions to sub-agents obscure harmful objectives, reducing robustness. Our results highlight the need for security-aware design in MAS. Link to code is https://github.com/microsoft/SafeAgents
Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 16 of its 58 references that are in the map. First posted on arXiv 14 Nov 2025.
Builds on
16 of 58 references.
Closest ancestors · 9
Same direction, newest first
- 2025ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoning26 Mar
- 2025SafeArena: Evaluating the Safety of Autonomous Web Agents6 Mar
- 2024RedCode: Risky Code Execution and Generation Benchmark for Code Agents12 Nov
- 2024Refusal-Trained LLMs Are Easily Jailbroken As Browser Agents11 Oct
- 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents11 Oct
- 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases17 Jul
- 2024Dissecting Adversarial Robustness of Multimodal LM Agents18 Jun
- 2024AutoDefense: Multi-Agent LLM Defense against Jailbreak Attacks2 Mar
- 2023Identifying the Risks of LM Agents with an LM-Emulated Sandbox25 Sep
Other references · 7
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2017Deep reinforcement learning from human preferencescited by 3,846
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
- 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
- 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 460
- 2023Are aligned neural networks adversarially aligned?cited by 305
- 2025Multi-Agent Security Tax: Trading Off Security and Collaboration Capabilities in Multi-Agent Systemscited by 27
Built on by
Citing papers per year: 2026: 8.
8 papers.
- 2026Learning When to Act or Refuse: Guarding Agentic Reasoning Models for Safe Multi-Step Tool Usecited by 7
- 2026Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluationcited by 5
- 2026Semantic Intent Fragmentation: A Single-Shot Compositional Attack on Multi-Agent AI Pipelinescited by 4
- 2026RvB: Automating AI System Hardening via Iterative Red-Blue Gamescited by 2
- 2026Taxonomy and Consistency Analysis of Safety Benchmarks for AI Agentscited by 1
- 2026SoK: When Safe Agents Fail Together: The Security of Multi Agent LLM Systemscited by 1
- 2026Adversarial Stress Testing of Role-Playing Language Agents using Multi-Agent Evaluationcited by 0
- 2026Trust the Brand, Lose Control: How Identity Hijacks LLM Agent Orchestrationcited by 0
- 2026Trust the Brand, Lose Control: How Identity Hijacks LLM Agent Orchestration26 Sep 2026
- 2026SoK: When Safe Agents Fail Together: The Security of Multi Agent LLM Systems1 Sep 2026
- 2026Adversarial Stress Testing of Role-Playing Language Agents using Multi-Agent Evaluation4 Aug 2026
- 2026Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluation9 Jun 2026
- 2026Taxonomy and Consistency Analysis of Safety Benchmarks for AI Agents11 Apr 2026
- 2026Semantic Intent Fragmentation: A Single-Shot Compositional Attack on Multi-Agent AI Pipelines8 Apr 2026
- 2026Learning When to Act or Refuse: Guarding Agentic Reasoning Models for Safe Multi-Step Tool Use3 Mar 2026
- 2026RvB: Automating AI System Hardening via Iterative Red-Blue Games27 Jan 2026
Similar, not cited
Close in wording, no citation either way
- 2026Adversarial Attacks in Multi-Agent LLM Pipelines: Unveiling Structural Vulnerabilities in Agentic AI Architectures
- 2025TAMAS: Benchmarking Adversarial Risks in Multi-Agent LLM Systems
- 2026DUMA-Bench: A Dual-Control Multi-Agent Benchmark for Evaluating LLM Agent Security
- 2026Conjunctive Prompt Attacks in Multi-Agent LLM Systems
- 2024AgentMonitor: A Plug-and-Play Framework for Predictive and Secure Multi-Agent Systems
- 2025Breaking Agent Backbones: Evaluating the Security of Backbone LLMs in AI Agents
- 2025OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safety
- 2025Multi-Agent Systems Execute Arbitrary Malicious Code