mlsanityData to 7 Oct 2026

AutoDefense: Multi-Agent LLM Defense against Jailbreak Attacks

arXiv 2403.04783 ↗First posted 2 Mar 2024cs.LG cs.CL cs.CRcited by 135

On the map: LLM agents and tool use → Agent threats, guardrails and governance

also in 1 more
Abstract (arXiv)

Despite extensive pre-training in moral alignment to prevent generating harmful information, large language models (LLMs) remain vulnerable to jailbreak attacks. In this paper, we propose AutoDefense, a multi-agent defense framework that filters harmful responses from LLMs. With the response-filtering mechanism, our framework is robust against different jailbreak attack prompts, and can be used to defend different victim models. AutoDefense assigns different roles to LLM agents and employs them to complete the defense task collaboratively. The division in tasks enhances the overall instruction-following of LLMs and enables the integration of other defense components as tools. With AutoDefense, small open-source LMs can serve as agents and defend larger models against jailbreak attacks. Our experiments show that AutoDefense can effectively defense against different jailbreak attacks, while maintaining the performance at normal user request. For example, we reduce the attack success rate on GPT-3.5 from 55.74% to 7.95% using LLaMA-2-13b with a 3-agent system. Our code and data are publicly available at https://github.com/XHMY/AutoDefense.

Key facts. Cited by 135 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 40 of its 57 references that are in the map. First posted on arXiv 2 Mar 2024.

Builds on

40 of 57 references.

References · 40

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  7. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  8. 2024Mixtral of Expertscited by 1,785
  9. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
  10. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  11. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  12. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
28 more
  1. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  2. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  3. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  4. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 995
  5. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  6. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  7. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 554
  8. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 478
  9. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 434
  10. 2023Detecting Language Model Attacks with Perplexitycited by 411
  11. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 401
  12. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreakercited by 337
  13. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 320
  14. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 226
  15. 2023Multilingual Jailbreak Challenges in Large Language Modelscited by 222
  16. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 187
  17. 2019Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attackcited by 176
  18. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLMcited by 173
  19. 2023The Capacity for Moral Self-Correction in Large Language Modelscited by 171
  20. 2023Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritizationcited by 156
  21. 2024The Impact of Reasoning Step Length on Large Language Modelscited by 152
  22. 2021Anticipating Safety Issues in E2E Conversational AI: Framework and Toolingcited by 77
  23. 2023Attack Prompt Generation for Red Teaming and Defending Large Language Modelscited by 73
  24. 2024Defending LLMs against Jailbreaking Attacks via Backtranslationcited by 65
  25. 2023Black Box Adversarial Prompting for Foundation Modelscited by 62
  26. 2023Self-Guard: Empower the LLM to Safeguard Itselfcited by 52
  27. 2023Bergeron: Combating Adversarial Attacks through a Conscience-Based Alignment Frameworkcited by 29
  28. 2023Why do universal adversarial attacks work on large language models?: Geometry might be the answercited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 35; 2025: 68; 2026: 32.

135 papers.

Sort:

By area:

  1. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  2. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194
  3. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 122
  4. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 116
  5. 2025X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agentscited by 112
  6. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  7. 2025A Survey on Trustworthy LLM Agents: Threats and Countermeasurescited by 93
  8. 2024Guardrail Baselines for Unlearning in LLMscited by 90
  9. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 67
  10. 2025Multi-Agent Evolve: LLM Self-Improve through Co-evolutioncited by 58
  11. 2024Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMscited by 47
  12. 2025A Survey on the Optimization of Large Language Model-based Agentscited by 45
38 more
  1. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  2. 2024Jailbreaking and Mitigation of Vulnerabilities in Large Language Modelscited by 33
  3. 2024Crabs: Consuming Resource via Auto-generation for LLM-DoS Attack under Black-box Settingscited by 33
  4. 2025textitAgents Under Siege: Breaking Pragmatic Multi-Agent LLM Systems with Optimized Prompt Attackscited by 31
  5. 2024JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Modelscited by 28
  6. 2024BlueSuffix: Reinforced Blue Teaming for Vision-Language Models Against Jailbreak Attackscited by 28
  7. 2024Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligencecited by 27
  8. 2024Large Model Based Agents: State-of-the-Art, Cooperation Paradigms, Security and Privacy, and Future Trendscited by 26
  9. 2025Towards Agentic Recommender Systems in the Era of Multimodal Large Language Modelscited by 25
  10. 2025The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMscited by 24
  11. 2024Adversarial Tuning: Defending Against Jailbreak Attacks for LLMscited by 23
  12. 2025A Survey on the Safety and Security Threats of Computer-Using Agents: JARVIS or Ultron?cited by 23
  13. 2024When LLMs Go Online: The Emerging Threat of Web-Enabled LLMscited by 22
  14. 2024Mission Impossible: A Statistical Perspective on Jailbreaking LLMscited by 21
  15. 2025EmoAgent: Assessing and Safeguarding Human-AI Interaction for Mental Health Safetycited by 21
  16. 2024PAPILLON: Efficient and Stealthy Fuzz Testing-Powered Jailbreaks for LLMscited by 20
  17. 2025X-Boundary: Establishing Exact Safety Boundary to Shield LLMs from Multi-Turn Jailbreaks without Compromising Usabilitycited by 20
  18. 2025Silent Leaks: Implicit Knowledge Extraction Attack on RAG Systems through Benign Queriescited by 19
  19. 2025Aegis: Automated Error Generation and Attribution for Multi-Agent Systemscited by 19
  20. 2025CORBA: Contagious Recursive Blocking Attacks on Multi-Agent Systems Based on Large Language Modelscited by 18
  21. 2025Revisiting Multi-Agent Debate as Test-Time Scaling: A Systematic Study of Conditional Effectivenesscited by 18
  22. 2024VLMGuard: Bootstrapping Malicious Prompt Detectors from Unlabeled Vision-Language Prompts in the Wildcited by 17
  23. 2025T2VShield: Model-Agnostic Jailbreak Defense for Text-to-Video Modelscited by 17
  24. 2025Security of Internet of Agents: Attacks and Countermeasurescited by 17
  25. 2025SoK: Evaluating Jailbreak Guardrails for Large Language Modelscited by 17
  26. 2024LLM-RankFusion: Mitigating Intrinsic Inconsistency in LLM-based Rankingcited by 16
  27. 2024LLM Whisperer: An Inconspicuous Attack to Bias LLM Responsescited by 15
  28. 2024Robust Neural Information Retrieval: An Adversarial and Out-of-distribution Perspectivecited by 15
  29. 2025ExCyTIn-Bench: Evaluating LLM agents on Cyber Threat Investigationcited by 15
  30. 2025MIND: A Multi-agent Framework for Zero-shot Harmful Meme Detectioncited by 14
  31. 2025BlockA2A: Towards Secure and Verifiable Agent-to-Agent Interoperabilitycited by 14
  32. 2026Beyond Max Tokens: Stealthy Resource Amplification via Tool Calling Chains in LLM Agentscited by 14
  33. 2025Dialogue Injection Attack: Jailbreaking LLMs through Context Manipulationcited by 13
  34. 2025X-Guard: Multilingual Guard Agent for Content Moderationcited by 13
  35. 2025Attack and defense techniques in large language models: A survey and new perspectivescited by 13
  36. 2025LLM Jailbreak Detection for (Almost) Free!cited by 13
  37. 2025DECEPTICON: How Dark Patterns Manipulate Web Agentscited by 13
  38. 2026Agent-Sentry: Bounding LLM Agents via Execution Provenancecited by 13

50 most cited of 135 All 135 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025From LLMs to MLLMs to Agents: A Survey of Emerging Paradigms in Jailbreak Attacks and Defenses within LLM Ecosystem
  2. 2025PandaGuard: Systematic Evaluation of LLM Safety against Jailbreaking Attacks
  3. 2024SelfDefend: LLMs Can Defend Themselves against Jailbreaking in a Practical Manner
  4. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents
  5. 2024Intention Analysis Makes LLMs A Good Jailbreak Defender
  6. 2025A Multi-Agent LLM Defense Pipeline Against Prompt Injection Attacks
  7. 2025MetaCipher: A Time-Persistent and Universal Multi-Agent Framework for Cipher-Based Jailbreak Attacks for LLMs
  8. 2024JAILJUDGE: A Comprehensive Jailbreak Judge Benchmark with Multi-Agent Enhanced Explanation Evaluation Framework