Amplified Vulnerabilities: Structured Jailbreak Attacks on LLM-based Multi-Agent Debate
On the map: LLM reasoning → Reasoning for agents, planning and code
also in 2 more
Abstract (arXiv)
Multi-Agent Debate (MAD), leveraging collaborative interactions among Large Language Models (LLMs), aim to enhance reasoning capabilities in complex tasks. However, the security implications of their iterative dialogues and role-playing characteristics, particularly susceptibility to jailbreak attacks eliciting harmful content, remain critically underexplored. This paper systematically investigates the jailbreak vulnerabilities of four prominent MAD frameworks built upon leading commercial LLMs (GPT-4o, GPT-4, GPT-3.5-turbo, and DeepSeek) without compromising internal agents. We introduce a novel structured prompt-rewriting framework specifically designed to exploit MAD dynamics via narrative encapsulation, role-driven escalation, iterative refinement, and rhetorical obfuscation. Our extensive experiments demonstrate that MAD systems are inherently more vulnerable than single-agent setups. Crucially, our proposed attack methodology significantly amplifies this fragility, increasing average harmfulness from 28.14% to 80.34% and achieving attack success rates as high as 80% in certain scenarios. These findings reveal intrinsic vulnerabilities in MAD architectures and underscore the urgent need for robust, specialized defenses prior to real-world deployment.
Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 45 of its 48 references that are in the map. First posted on arXiv 23 Apr 2025.
Builds on
45 of 48 references.
Closest ancestors · 6
Same direction, newest first
- 2025Stop Overvaluing Multi-Agent Debate -- We Must Rethink Evaluation and Embrace Model Heterogeneity12 Feb
- 2025S2-MAD: Breaking the Token Barrier to Enhance Multi-Agent Debate Efficiency7 Feb
- 2024GroupDebate: Enhancing the Efficiency of Multi-Agent Debate Using Group Discussion21 Sep
- 2024Improving Multi-Agent Debate with Sparse Communication Topology17 Jun
- 2023Learning to Break: Knowledge-Enhanced Reasoning in Multi-Agent Debate System8 Dec
- 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate30 May
Other references · 39
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
- 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
- 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
- 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
- 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
- 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 785
- 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 742
- 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviorscited by 598
- 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 489
27 more
- 2023Detecting Language Model Attacks with Perplexitycited by 431
- 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 423
- 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 387
- 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 278
- 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 230
- 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulationcited by 214
- 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 191
- 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fastcited by 153
- 2024Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systemscited by 133
- 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studiescited by 131
- 2024On Prompt-Driven Safeguarding for Large Language Modelscited by 116
- 2024How Alignment and Jailbreak Work: Explain LLM Safety through Intermediate Hidden Statescited by 98
- 2024GradSafe: Detecting Jailbreak Prompts for LLMs via Safety-Critical Gradient Analysiscited by 96
- 2024Internet of Agents: Weaving a Web of Heterogeneous Agents for Collaborative Intelligencecited by 86
- 2024Flooding Spread of Manipulated Knowledge in LLM-Based Multi-Agent Communitiescited by 86
- 2023Exchange-of-Thought: Enhancing Large Language Model Capabilities through Cross-Model Communicationcited by 82
- 2024Breaking Agents: Compromising Autonomous LLM Agents Through Malfunction Amplificationcited by 81
- 2024PsySafe: A Comprehensive Framework for Psychological-based Attack, Defense, and Evaluation of Multi-agent System Safetycited by 80
- 2023Attack Prompt Generation for Red Teaming and Defending Large Language Modelscited by 76
- 2024CodeChameleon: Personalized Encryption Framework for Jailbreaking Large Language Modelscited by 76
- 2024Play Guessing Game with LLM: Indirect Jailbreak Attack with Implicit Cluescited by 68
- 2024PRP: Propagating Universal Perturbations to Attack Large Language Model Guard-Railscited by 52
- 2025textitAgents Under Siege: Breaking Pragmatic Multi-Agent LLM Systems with Optimized Prompt Attackscited by 34
- 2024JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Modelscited by 31
- 2024Is poisoning a real threat to LLM alignment? Maybe more so than you thinkcited by 28
- 2024Foot In The Door: Understanding Large Language Model Jailbreaking via Cognitive Psychologycited by 27
- 2025Stay Focused: Problem Drift in Multi-Agent Debatecited by 23
Built on by
Citing papers per year: 2025: 4; 2026: 6.
10 papers.
- 2025Free-MAD: Consensus-Free Multi-Agent Debatecited by 29
- 2025A Survey on the Safety and Security Threats of Computer-Using Agents: JARVIS or Ultron?cited by 23
- 2025MAD-Spear: A Conformity-Driven Prompt Injection Attack on Multi-Agent Debate Systemscited by 10
- 2026From Logic Monopoly to Social Contract: Separation of Power and the Institutional Foundations for Autonomous Agent Economiescited by 3
- 2026Architecture Matters for Multi-Agent Securitycited by 3
- 2025SoK: A Comprehensive Security Analysis of Jailbreak Resilience in GPT and DeepSeek Modelscited by 1
- 2026SoK: When Safe Agents Fail Together: The Security of Multi Agent LLM Systemscited by 1
- 2026Governance by Design: A Parsonian Institutional Architecture for Internet-Wide Agent Societiescited by 0
- 2026Agent-Native Immune System: Architecture, Taxonomy, and Engineeringcited by 0
- 2026Toward Secure AI-Powered Penetration Testing Agents: Security Threats, Guardrails, and Architectural Perspectivescited by 0
- 2026Toward Secure AI-Powered Penetration Testing Agents: Security Threats, Guardrails, and Architectural Perspectives15 Sep 2026
- 2026SoK: When Safe Agents Fail Together: The Security of Multi Agent LLM Systems1 Sep 2026
- 2026Agent-Native Immune System: Architecture, Taxonomy, and Engineering26 Jun 2026
- 2026Architecture Matters for Multi-Agent Security25 Apr 2026
- 2026Governance by Design: A Parsonian Institutional Architecture for Internet-Wide Agent Societies13 Apr 2026
- 2026From Logic Monopoly to Social Contract: Separation of Power and the Institutional Foundations for Autonomous Agent Economies26 Mar 2026
- 2025Free-MAD: Consensus-Free Multi-Agent Debate14 Sep 2025
- 2025MAD-Spear: A Conformity-Driven Prompt Injection Attack on Multi-Agent Debate Systems17 Jul 2025
- 2025SoK: A Comprehensive Security Analysis of Jailbreak Resilience in GPT and DeepSeek Models23 Jun 2025
- 2025A Survey on the Safety and Security Threats of Computer-Using Agents: JARVIS or Ultron?16 May 2025
Similar, not cited
Close in wording, no citation either way
- 2025Echoes of Human Malice in Agents: Benchmarking LLMs for Multi-Turn Online Harassment Attacks
- 2024MRJ-Agent: An Effective Jailbreak Agent for Multi-Round Dialogue
- 2025Evaluating Multi-Agent Defences Against Jailbreaking Attacks on Large Language Models
- 2025PLAGUE: Plug-and-play framework for Lifelong Adaptive Generation of Multi-turn Exploits
- 2026MADBench: Benchmarking the Security of Multi-Agent Debate
- 2025Chain-of-Lure: A Universal Jailbreak Attack Framework using Unconstrained Synthetic Narratives
- 2026On Understanding, Identifying, and Mitigating Vulnerabilities in Agentic Large Language Models
- 2026A Self-Evolving Multi-Agent Framework Defense against LLM Jailbreak Attacks