mlsanityData to 7 Oct 2026

JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Models

arXiv 2407.01599 ↗First posted 26 Jun 2024cs.CL cs.CR cs.CVcited by 67

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

The rapid evolution of artificial intelligence (AI) through developments in Large Language Models (LLMs) and Vision-Language Models (VLMs) has brought significant advancements across various technological domains. While these models enhance capabilities in natural language processing and visual interactive tasks, their growing adoption raises critical concerns regarding security and ethical alignment. This survey provides an extensive review of the emerging field of jailbreaking--deliberately circumventing the ethical and operational boundaries of LLMs and VLMs--and the consequent development of defense mechanisms. Our study categorizes jailbreaks into seven distinct types and elaborates on defense strategies that address these vulnerabilities. Through this comprehensive examination, we identify research gaps and propose directions for future studies to enhance the security frameworks of LLMs and VLMs. Our findings underscore the necessity for a unified perspective that integrates both jailbreak strategies and defensive solutions to foster a robust, secure, and reliable environment for the next generation of language models. More details can be found on our website: https://chonghan-chen.com/llm-jailbreak-zoo-survey/.

Key facts. Cited by 67 papers in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 140 of its 161 references that are in the map. First posted on arXiv 26 Jun 2024.

Builds on

140 of 161 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks3 Apr
  2. 2024An Image Is Worth 1000 Lies: Adversarial Transferability across Prompts on Vision-Language Models14 Mar
  3. 2024Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformation14 Mar
  4. 2024AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Prompting14 Mar
  5. 2024MLLM-Protector: Ensuring MLLM's Safety without Hurting Performance5 Jan
  6. 2023OT-Attack: Enhancing Adversarial Transferability of Vision-Language Models via Optimal Transport Optimization7 Dec
  7. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models29 Nov
  8. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts9 Nov
  9. 2023How Robust is Google's Bard to Adversarial Image Attacks?21 Sep
  10. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models26 Jul
  11. 2023Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models26 Jul
  12. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Models22 Jun
2 more
  1. 2023On Evaluating Adversarial Robustness of Large Vision-Language Models26 May
  2. 2022Towards Adversarial Attack on Vision-Language Pre-training Models19 Jun

Other references · 126

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  10. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  11. 2017Deep reinforcement learning from human preferencescited by 3,846
  12. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
38 more
  1. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  2. 2020Learning to summarize from human feedbackcited by 2,562
  3. 2020Extracting Training Data from Large Language Modelscited by 2,144
  4. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  5. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  6. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  7. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  8. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  9. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  10. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  11. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 738
  12. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  13. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  14. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  15. 2018Improving Transferability of Adversarial Examples with Input Diversitycited by 591
  16. 2016Delving into Transferable Adversarial Examples and Black-box Attackscited by 530
  17. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  18. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
  19. 2016Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samplescited by 439
  20. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  21. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  22. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 434
  23. 2020Backdoor Learning: A Surveycited by 412
  24. 2023Detecting Language Model Attacks with Perplexitycited by 411
  25. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 401
  26. 2018AI safety via debatecited by 384
  27. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
  28. 2024A StrongREJECT for Empty Jailbreakscited by 352
  29. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreakercited by 337
  30. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 320
  31. 2023Are aligned neural networks adversarially aligned?cited by 305
  32. 2023Low-Resource Languages Jailbreak GPT-4cited by 290
  33. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  34. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 268
  35. 2023Certifying LLM Safety against Adversarial Promptingcited by 248
  36. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easilycited by 242
  37. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decodingcited by 233
  38. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 226

50 most cited of 126 All 126 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 37; 2026: 15.

67 papers.

Sort:

By area:

  1. 2025A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluationscited by 66
  2. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  3. 2024ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Timecited by 57
  4. 2024Steering Away from Harm: An Adaptive Approach to Defending Vision Language Model Against Jailbreakscited by 44
  5. 2025VPI-Bench: Visual Prompt Injection Attacks for Computer-Use Agentscited by 42
  6. 2024Chain of Attack: On the Robustness of Vision-Language Models Against Transfer-Based Adversarial Attackscited by 37
  7. 2025Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilitiescited by 36
  8. 2024Jailbreak Attacks and Defenses against Multimodal Generative Models: A Surveycited by 31
  9. 2025Distraction is All You Need for Multimodal Large Language Model Jailbreakingcited by 31
  10. 2025RAG LLMs are Not Safer: A Safety Analysis of Retrieval-Augmented Generation for Large Language Modelscited by 23
  11. 2025Visual Contextual Attack: Jailbreaking MLLMs with Image-Driven Context Injectioncited by 23
  12. 2024Jailbreak Antidote: Runtime Safety-Utility Balance via Sparse Representation Adjustment in Large Language Modelscited by 21
38 more
  1. 2024Surveying the MLLM Landscape: A Meta-Review of Current Surveyscited by 20
  2. 2024Differential Privacy Mechanisms in Neural Tangent Kernel Regressioncited by 14
  3. 2024Provable Differentially Private Computation of the Cross-Attention Mechanismcited by 14
  4. 2024Easier Said Than Done: Unpacking Intent-Behavior Gap in Jailbreaking LLM-Based Robotscited by 14
  5. 2025Speak Easy: Eliciting Harmful Jailbreaks from LLMs with Simple Interactionscited by 14
  6. 2025From LLMs to MLLMs to Agents: A Survey of Emerging Paradigms in Jailbreak Attacks and Defenses within LLM Ecosystemcited by 11
  7. 2024PathSeeker: Exploring LLM Security Vulnerabilities with a Reinforcement Learning-Based Jailbreak Approachcited by 9
  8. 2024The Dark Side of Trust: Authority Citation-Driven Jailbreak Attacks on Large Language Modelscited by 9
  9. 2024Safe + Safe = Unsafe? Exploring How Safe Images Can Be Exploited to Jailbreak Large Vision-Language Modelscited by 9
  10. 2025Robust-LLaVA: On the Effectiveness of Large-Scale Robust Image Encoders for Multi-modal Large Language Modelscited by 9
  11. 2025GuidedBench: Measuring and Mitigating the Evaluation Discrepancies of In-the-wild LLM Jailbreak Methodscited by 9
  12. 2025The Singapore Consensus on Global AI Safety Research Prioritiescited by 9
  13. 2025RedCodeAgent: Automatic Red-teaming Agent against Diverse Code Agentscited by 9
  14. 2024Blockchain for Large Language Model Security and Safety: A Holistic Surveycited by 8
  15. 2025Mixed Signals: Decoding VLMs' Reasoning and Underlying Bias in Vision-Language Conflictcited by 7
  16. 2025DAVSP: Safety Alignment for Large Vision-Language Models via Deep Aligned Visual Safety Promptcited by 7
  17. 2025Attention Eclipse: Manipulating Attention to Bypass LLM Safety-Alignmentcited by 6
  18. 2025When Style Breaks Safety: Defending LLMs Against Superficial Style Alignmentcited by 6
  19. 2025Learning to Conceal Risk: Controllable Multi-turn Red Teaming for LLMs in the Financial Domaincited by 6
  20. 2026HoneyTrap: Deceiving Large Language Model Attackers to Honeypot Traps with Resilient Multi-Agent Defensecited by 6
  21. 2024BlackDAN: A Black-Box Multi-Objective Approach for Effective and Contextual Jailbreaking of Large Language Modelscited by 5
  22. 2025CoBia: Constructed Conversations Can Trigger Otherwise Concealed Societal Biases in LLMscited by 5
  23. 2025SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defensescited by 4
  24. 2024Harnessing Task Overload for Scalable Jailbreak Attacks on Large Language Modelscited by 3
  25. 2025Safeguarding Vision-Language Models: Mitigating Vulnerabilities to Gaussian Noise in Perturbation-based Attackscited by 3
  26. 2026Controlling Output Rankings in Generative Engines for LLM-based Searchcited by 3
  27. 2026WARD: Adversarially Robust Defense of Web Agents Against Prompt Injectionscited by 3
  28. 2026SCI-Defense: Defending Manipulation Attacks from Generative Engine Optimizationcited by 3
  29. 2025EigenShield: Causal Subspace Filtering via Random Matrix Theory for Adversarially Robust Vision-Language Modelscited by 2
  30. 2025From Evaluation to Defense: Advancing Safety in Video Large Language Modelscited by 2
  31. 2025From Hallucinations to Jailbreaks: Rethinking the Vulnerability of Large Foundation Modelscited by 2
  32. 2025Diffusion-based Cumulative Adversarial Purification for Vision Language Modelscited by 2
  33. 2025GuardVal: Dynamic Large Language Model Jailbreak Evaluation for Comprehensive Safety Testingcited by 2
  34. 2025Breaking Minds, Breaking Systems: Jailbreaking Large Language Models via Human-like Psychological Manipulationcited by 2
  35. 2026Red-teaming the Multimodal Reasoning: Jailbreaking Vision-Language Models via Cross-modal Entanglement Attackscited by 2
  36. 2026TreeTeaming: Autonomous Red-Teaming of Vision-Language Models via Hierarchical Strategy Explorationcited by 2
  37. 2025Linearly Decoding Refused Knowledge in Aligned Language Modelscited by 1
  38. 2025Learning to Detect Unseen Jailbreak Attacks in Large Vision-Language Modelscited by 1

50 most cited of 67 All 67 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024From LLMs to MLLMs: Exploring the Landscape of Multimodal Jailbreaking
  2. 2026SoK: Rethinking Jailbreaking in the Era of Agentic AI: Attacks, Defenses, and Practical Consideration
  3. 2024Understanding Jailbreak Success: A Study of Latent Space Dynamics in Large Language Models
  4. 2025Do Internal Layers of LLMs Reveal Patterns for Jailbreak Detection?
  5. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Survey
  6. 2025Do Methods to Jailbreak and Defend LLMs Generalize Across Languages?
  7. 2024Defending against Jailbreak through Early Exit Generation of Large Language Models
  8. 2025JailDAM: Jailbreak Detection with Adaptive Memory for Vision-Language Model