From LLMs to MLLMs to Agents: A Survey of Emerging Paradigms in Jailbreak Attacks and Defenses within LLM Ecosystem
On the map: LLM agents and tool use → Agent threats, guardrails and governance
also in 1 more
Abstract (arXiv)
Large language models (LLMs) are rapidly evolving from single-modal systems to multimodal LLMs and intelligent agents, significantly expanding their capabilities while introducing increasingly severe security risks. This paper presents a systematic survey of the growing complexity of jailbreak attacks and corresponding defense mechanisms within the expanding LLM ecosystem. We first trace the developmental trajectory from LLMs to MLLMs and Agents, highlighting the core security challenges emerging at each stage. Next, we categorize mainstream jailbreak techniques from both the attack impact and visibility perspectives, and provide a comprehensive analysis of representative attack methods, related datasets, and evaluation metrics. On the defense side, we organize existing strategies based on response timing and technical approach, offering a structured understanding of their applicability and implementation. Furthermore, we identify key limitations in existing surveys, such as insufficient attention to agent-specific security issues, the absence of a clear taxonomy for hybrid jailbreak methods, a lack of detailed analysis of experimental setups, and outdated coverage of recent advancements. To address these limitations, we provide an updated synthesis of recent work and outline future research directions in areas such as dataset construction, evaluation framework optimization, and strategy generalization. Our study seeks to enhance the understanding of jailbreak mechanisms and facilitate the advancement of more resilient and adaptive defense strategies in the context of ever more capable LLMs.
Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 133 of its 155 references that are in the map. First posted on arXiv 18 Jun 2025.
Builds on
133 of 155 references.
Closest ancestors · 16
Same direction, newest first
- 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment22 Apr
- 2025Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents27 Feb
- 2024MRJ-Agent: An Effective Jailbreak Agent for Multi-Round Dialogue6 Nov
- 2024Breaking ReAct Agents: Foot-in-the-Door Attack Will Get You In22 Oct
- 2024NetSafe: Exploring the Topological Safety of Multi-agent Networks21 Oct
- 2024Refusal-Trained LLMs Are Easily Jailbroken As Browser Agents11 Oct
- 2024JAILJUDGE: A Comprehensive Jailbreak Judge Benchmark with Multi-Agent Enhanced Explanation Evaluation Framework11 Oct
- 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents11 Oct
- 2024Breaking Agents: Compromising Autonomous LLM Agents Through Malfunction Amplification30 Jul
- 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studies28 Jul
- 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases17 Jul
- 2024Flooding Spread of Manipulated Knowledge in LLM-Based Multi-Agent Communities10 Jul
4 more
- 2024AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents19 Jun
- 2024The Wolf Within: Covert Injection of Malice into MLLM Societies via an MLLM Operative20 Feb
- 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fast13 Feb
- 2024PsySafe: A Comprehensive Framework for Psychological-based Attack, Defense, and Evaluation of Multi-agent System Safety22 Jan
Other references · 117
- 2017Attention Is All You Needcited by 60,361
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2023GPT-4 Technical Reportcited by 18,396
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2020Scaling Laws for Neural Language Modelscited by 6,586
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
- 2016Layer Normalizationcited by 5,669
38 more
- 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
- 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
- 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
- 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
- 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
- 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
- 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
- 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
- 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 785
- 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 769
- 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 672
- 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 651
- 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 573
- 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 560
- 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 536
- 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 505
- 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 498
- 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 489
- 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 431
- 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 423
- 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Promptscited by 398
- 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 387
- 2024A StrongREJECT for Empty Jailbreakscited by 385
- 2024StruQ: Defending Against Prompt Injection with Structured Queriescited by 375
- 2023DeepInception: Hypnotize Large Language Model to Be Jailbreakercited by 367
- 2023Visual Adversarial Examples Jailbreak Aligned Large Language Modelscited by 328
- 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 284
- 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Modelscited by 273
- 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 264
- 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easilycited by 259
- 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 244
- 2023Multilingual Jailbreak Challenges in Large Language Modelscited by 240
- 2024JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attackscited by 230
- 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 230
- 2025Survey on Evaluation of LLM-based Agentscited by 204
- 2023SafetyBench: Evaluating the Safety of Large Language Modelscited by 199
- 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 194
50 most cited of 117 All 117 (JSON)
Built on by
Citing papers per year: 2025: 2; 2026: 10.
12 papers.
- 2025SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defensescited by 5
- 2026Safer by Diffusion, Broken by Context: Diffusion LLM's Safety Blessing and Its Failure Modecited by 4
- 2026How Real is Your Jailbreak? Fine-grained Jailbreak Evaluation with Anchored Referencecited by 3
- 2025HAMSA: Hijacking Aligned Compact Models via Stealthy Automationcited by 2
- 2026Stop Fixating on Prompts: Reasoning Hijacking and Constraint Tightening for Red-Teaming LLM Agentscited by 2
- 2026MIDAS: Multi-Image Dispersion and Semantic Reconstruction for Jailbreaking MLLMscited by 1
- 2026SoK: Intent-Oriented Systematization of Multi-Turn LLM Jailbreakscited by 1
- 2026KC-Bench: A Dynamic Interactive Benchmark for Evaluating Knowledge Conflicts in LLM Agentscited by 1
- 2026Evaluating Proactive Risk Awareness of Large Language Modelscited by 0
- 2026Why Do Large Language Models Generate Harmful Content?cited by 0
- 2026Reasoning-targeted Jailbreak Attacks on Large Reasoning Models via Semantic Triggers and Psychological Framingcited by 0
- 2026SynChain: Inducing Computer-Use Agent Systems to Construct Their Own Attack Chainscited by 0
- 2026KC-Bench: A Dynamic Interactive Benchmark for Evaluating Knowledge Conflicts in LLM Agents3 Sep 2026
- 2026SynChain: Inducing Computer-Use Agent Systems to Construct Their Own Attack Chains7 Aug 2026
- 2026SoK: Intent-Oriented Systematization of Multi-Turn LLM Jailbreaks2 Aug 2026
- 2026Reasoning-targeted Jailbreak Attacks on Large Reasoning Models via Semantic Triggers and Psychological Framing17 Apr 2026
- 2026Why Do Large Language Models Generate Harmful Content?13 Apr 2026
- 2026Stop Fixating on Prompts: Reasoning Hijacking and Constraint Tightening for Red-Teaming LLM Agents7 Apr 2026
- 2026MIDAS: Multi-Image Dispersion and Semantic Reconstruction for Jailbreaking MLLMs28 Feb 2026
- 2026Evaluating Proactive Risk Awareness of Large Language Models24 Feb 2026
- 2026Safer by Diffusion, Broken by Context: Diffusion LLM's Safety Blessing and Its Failure Mode30 Jan 2026
- 2026How Real is Your Jailbreak? Fine-grained Jailbreak Evaluation with Anchored Reference4 Jan 2026
- 2025SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defenses17 Oct 2025
- 2025HAMSA: Hijacking Aligned Compact Models via Stealthy Automation22 Aug 2025
Similar, not cited
Close in wording, no citation either way
- 2025Evaluating Multi-Agent Defences Against Jailbreaking Attacks on Large Language Models
- 2025PandaGuard: Systematic Evaluation of LLM Safety against Jailbreaking Attacks
- 2026SoK: Rethinking Jailbreaking in the Era of Agentic AI: Attacks, Defenses, and Practical Consideration
- 2025Evolving Security in LLMs: A Study of Jailbreak Attacks and Defenses
- 2024Jailbreaking and Mitigation of Vulnerabilities in Large Language Models
- 2024A Comprehensive Study of Jailbreak Attack versus Defense for Large Language Models
- 2024Towards Understanding Jailbreak Attacks in LLMs: A Representation Space Analysis
- 2026Analysis of LLMs Against Prompt Injection and Jailbreak Attacks