mlsanityData to 14 Oct 2026

From LLMs to MLLMs to Agents: A Survey of Emerging Paradigms in Jailbreak Attacks and Defenses within LLM Ecosystem

arXiv 2506.15170 ↗First posted 18 Jun 2025cs.CRcited by 12

On the map: LLM agents and tool use → Agent threats, guardrails and governance

also in 1 more
Abstract (arXiv)

Large language models (LLMs) are rapidly evolving from single-modal systems to multimodal LLMs and intelligent agents, significantly expanding their capabilities while introducing increasingly severe security risks. This paper presents a systematic survey of the growing complexity of jailbreak attacks and corresponding defense mechanisms within the expanding LLM ecosystem. We first trace the developmental trajectory from LLMs to MLLMs and Agents, highlighting the core security challenges emerging at each stage. Next, we categorize mainstream jailbreak techniques from both the attack impact and visibility perspectives, and provide a comprehensive analysis of representative attack methods, related datasets, and evaluation metrics. On the defense side, we organize existing strategies based on response timing and technical approach, offering a structured understanding of their applicability and implementation. Furthermore, we identify key limitations in existing surveys, such as insufficient attention to agent-specific security issues, the absence of a clear taxonomy for hybrid jailbreak methods, a lack of detailed analysis of experimental setups, and outdated coverage of recent advancements. To address these limitations, we provide an updated synthesis of recent work and outline future research directions in areas such as dataset construction, evaluation framework optimization, and strategy generalization. Our study seeks to enhance the understanding of jailbreak mechanisms and facilitate the advancement of more resilient and adaptive defense strategies in the context of ever more capable LLMs.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 133 of its 155 references that are in the map. First posted on arXiv 18 Jun 2025.

Builds on

133 of 155 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment22 Apr
  2. 2025Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents27 Feb
  3. 2024MRJ-Agent: An Effective Jailbreak Agent for Multi-Round Dialogue6 Nov
  4. 2024Breaking ReAct Agents: Foot-in-the-Door Attack Will Get You In22 Oct
  5. 2024NetSafe: Exploring the Topological Safety of Multi-agent Networks21 Oct
  6. 2024Refusal-Trained LLMs Are Easily Jailbroken As Browser Agents11 Oct
  7. 2024JAILJUDGE: A Comprehensive Jailbreak Judge Benchmark with Multi-Agent Enhanced Explanation Evaluation Framework11 Oct
  8. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents11 Oct
  9. 2024Breaking Agents: Compromising Autonomous LLM Agents Through Malfunction Amplification30 Jul
  10. 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studies28 Jul
  11. 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases17 Jul
  12. 2024Flooding Spread of Manipulated Knowledge in LLM-Based Multi-Agent Communities10 Jul
4 more
  1. 2024AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents19 Jun
  2. 2024The Wolf Within: Covert Injection of Malice into MLLM Societies via an MLLM Operative20 Feb
  3. 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fast13 Feb
  4. 2024PsySafe: A Comprehensive Framework for Psychological-based Attack, Defense, and Evaluation of Multi-agent System Safety22 Jan

Other references · 117

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2020Language Models are Few-Shot Learnerscited by 32,419
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  7. 2023GPT-4 Technical Reportcited by 18,396
  8. 2022Training language models to follow instructions with human feedbackcited by 15,971
  9. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  10. 2020Scaling Laws for Neural Language Modelscited by 6,586
  11. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  12. 2016Layer Normalizationcited by 5,669
38 more
  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  6. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  8. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  9. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  10. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 785
  11. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 769
  12. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 672
  13. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 651
  14. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 573
  15. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 560
  16. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 536
  17. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 505
  18. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 498
  19. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 489
  20. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 431
  21. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 423
  22. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Promptscited by 398
  23. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 387
  24. 2024A StrongREJECT for Empty Jailbreakscited by 385
  25. 2024StruQ: Defending Against Prompt Injection with Structured Queriescited by 375
  26. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreakercited by 367
  27. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Modelscited by 328
  28. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 284
  29. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Modelscited by 273
  30. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 264
  31. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easilycited by 259
  32. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 244
  33. 2023Multilingual Jailbreak Challenges in Large Language Modelscited by 240
  34. 2024JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attackscited by 230
  35. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 230
  36. 2025Survey on Evaluation of LLM-based Agentscited by 204
  37. 2023SafetyBench: Evaluating the Safety of Large Language Modelscited by 199
  38. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 194

50 most cited of 117 All 117 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 10.

12 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2025Evaluating Multi-Agent Defences Against Jailbreaking Attacks on Large Language Models
  2. 2025PandaGuard: Systematic Evaluation of LLM Safety against Jailbreaking Attacks
  3. 2026SoK: Rethinking Jailbreaking in the Era of Agentic AI: Attacks, Defenses, and Practical Consideration
  4. 2025Evolving Security in LLMs: A Study of Jailbreak Attacks and Defenses
  5. 2024Jailbreaking and Mitigation of Vulnerabilities in Large Language Models
  6. 2024A Comprehensive Study of Jailbreak Attack versus Defense for Large Language Models
  7. 2024Towards Understanding Jailbreak Attacks in LLMs: A Representation Space Analysis
  8. 2026Analysis of LLMs Against Prompt Injection and Jailbreak Attacks