mlsanityData to 14 Oct 2026

Distraction is All You Need for Multimodal Large Language Model Jailbreaking

arXiv 2502.10794 ↗First posted 15 Feb 2025cs.CVcited by 36

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

Multimodal Large Language Models (MLLMs) bridge the gap between visual and textual data, enabling a range of advanced applications. However, complex internal interactions among visual elements and their alignment with text can introduce vulnerabilities, which may be exploited to bypass safety mechanisms. To address this, we analyze the relationship between image content and task and find that the complexity of subimages, rather than their content, is key. Building on this insight, we propose the Distraction Hypothesis, followed by a novel framework called Contrasting Subimage Distraction Jailbreaking (CS-DJ), to achieve jailbreaking by disrupting MLLMs alignment through multi-level distraction strategies. CS-DJ consists of two components: structured distraction, achieved through query decomposition that induces a distributional shift by fragmenting harmful prompts into sub-queries, and visual-enhanced distraction, realized by constructing contrasting subimages to disrupt the interactions among visual elements within the model. This dual strategy disperses the model's attention, reducing its ability to detect and mitigate harmful content. Extensive experiments across five representative scenarios and four popular closed-source MLLMs, including GPT-4o-mini, GPT-4o, GPT-4V, and Gemini-1.5-Flash, demonstrate that CS-DJ achieves average success rates of 52.40% for the attack success rate and 74.10% for the ensemble attack success rate. These results reveal the potential of distraction-based approaches to exploit and bypass MLLMs' defenses, offering new insights for attack strategies.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 42 of its 48 references that are in the map. First posted on arXiv 15 Feb 2025.

Builds on

42 of 48 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024Arondight: Red Teaming Large Vision Language Models with Auto-generated Multi-modal Jailbreak Prompts21 Jul
  2. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Models26 Jun
  3. 2024White-box Multimodal Jailbreaks Against Large Vision-Language Models28 May
  4. 2024Images are Achilles' Heel of Alignment: Exploiting Visual Vulnerabilities for Jailbreaking Multimodal Large Language Models14 Mar
  5. 2024ImgTrojan: Jailbreaking Vision-Language Models with ONE Image5 Mar
  6. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models29 Nov
  7. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts9 Nov
  8. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models26 Jul
  9. 2023Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models26 Jul
  10. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Models22 Jun
  11. 2023On Evaluating Adversarial Robustness of Large Vision-Language Models26 May

Other references · 31

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2020Denoising Diffusion Implicit Modelscited by 9,781
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2024GPT-4o System Cardcited by 4,588
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  9. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  10. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  11. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
19 more
  1. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  2. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,114
  3. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  4. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  5. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 360
  6. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 300
  7. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 290
  8. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 232
  9. 2023Prompting Large Language Models with Speech Recognition Abilitiescited by 140
  10. 2024Jailbreaking Attack against Multimodal Large Language Modelcited by 135
  11. 2023HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Datacited by 126
  12. 2023How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMscited by 108
  13. 2024Making Them Ask and Answer: Jailbreaking Large Language Models in Few Queries via Disguise and Reconstructioncited by 107
  14. 2023AdvCLIP: Downstream-agnostic Adversarial Examples in Multimodal Contrastive Learningcited by 103
  15. 2024Visual-RolePlay: Universal Jailbreak Attack on MultiModal Large Language Models via Role-playing Image Charactercited by 68
  16. 2023Jailbreaking GPT-4V via Self-Adversarial Attacks with System Promptscited by 65
  17. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Taskscited by 57
  18. 2024Harnessing the Power of MLLMs for Transferable Text-to-Image Person ReIDcited by 39
  19. 2024BaThe: Defense against the Jailbreak Attack in Multimodal Large Language Models by Treating Harmful Instruction as Backdoor Triggercited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 14; 2026: 22.

36 papers.

Sort:

By area:

  1. 2025A Survey of LLM-Driven AI Agent Communication: Protocols, Security Risks, and Defense Countermeasurescited by 57
  2. 2025Video-SafetyBench: A Benchmark for Safety Evaluation of Video LVLMscited by 28
  3. 2025Pushing the Limits of Safety: A Technical Report on the ATLAS Challenge 2025cited by 12
  4. 2025FORCE: Transferable Visual Jailbreaking Attacks via Feature Over-Reliance CorrEctioncited by 11
  5. 2025OmniSafeBench-MM: A Unified Benchmark and Toolbox for Multimodal Jailbreak Attack-Defense Evaluationcited by 11
  6. 2025Implicit Jailbreak Attacks via Cross-Modal Information Concealment on Vision-Language Modelscited by 6
  7. 2025Towards Effective MLLM Jailbreaking Through Balanced On-Topicness and OOD-Intensitycited by 4
  8. 2025GuardTrace-VL: Detecting Unsafe Multimodel Reasoning via Iterative Safety Supervisioncited by 4
  9. 2026Adversarial Attacks on Multimodal Large Language Models: A Comprehensive Surveycited by 4
  10. 2026TreeTeaming: Autonomous Red-Teaming of Vision-Language Models via Hierarchical Strategy Explorationcited by 3
  11. 2025Probabilistic Modeling of Jailbreak on Multimodal LLMs: From Quantification to Applicationcited by 2
  12. 2025Watch, Listen, Understand, Mislead: Tri-modal Adversarial Attacks on Short Videos for Content Appropriateness Evaluationcited by 2
24 more
  1. 2025JaiLIP: Jailbreaking Vision-Language Models via Loss Guided Image Perturbationcited by 2
  2. 2026Beyond Visual Safety: Jailbreaking Multimodal Large Language Models for Harmful Image Generation via Semantic-Agnostic Inputscited by 2
  3. 2026When the Prompt Becomes Visual: Vision-Centric Jailbreak Attacks for Large Image Editing Modelscited by 2
  4. 2025VERA-V: Variational Inference Framework for Jailbreaking Vision-Language Modelscited by 1
  5. 2025Beyond Text: Multimodal Jailbreaking of Vision-Language and Audio Models through Perceptually Simple Transformationscited by 1
  6. 2025Why Does Weak-OOD Help? A Further Step Towards Understanding Jailbreaking VLMscited by 1
  7. 2026PC2: Politically Controversial Content Generation via Jailbreaking Attacks on GPT-based Text-to-Image Modelscited by 1
  8. 2026Text is All You Need for Vision-Language Model Jailbreakingcited by 1
  9. 2026Robustness of Vision Language Models Against Split-Image Harmful Input Attackscited by 1
  10. 2026MIDAS: Multi-Image Dispersion and Semantic Reconstruction for Jailbreaking MLLMscited by 1
  11. 2026DMN: A Compositional Framework for Jailbreaking Multimodal LLMs with Multi-Image Inputscited by 1
  12. 2026VPA-Guard: Defending and Benchmarking Image-to-Video Generation Against Visual Prompt Attackscited by 1
  13. 2026Structured Visual Narratives Undermine Safety Alignment in Multimodal Large Language Modelscited by 0
  14. 2026Seeing No Evil: Blinding Large Vision-Language Models to Safety Instructions via Adversarial Attention Hijackingcited by 0
  15. 2026Conceal, Reconstruct, Jailbreak: Exploiting the Reconstruction-Concealment Tradeoff in MLLMscited by 0
  16. 2026New Wide-Net-Casting Jailbreak Attacks Risk Large Modelscited by 0
  17. 2026Same Model, Different Weakness: How Language and Modality Reshape the Jailbreak Attack Surface in Frontier MLLMscited by 0
  18. 2026Benign Inputs, Harmful Outputs: Cross-Modal Jailbreaking via Distributed Semantic Recompositioncited by 0
  19. 2026Jailbreaking Multimodal Large Language Models using Multi-Clip Videocited by 0
  20. 2026RedEdit: Agentic Red-Teaming of Image Safety Classifiers via MCTS-Guided Photo-Editingcited by 0
  21. 2026PHANTOM: A Large-Scale Dataset of Multimodal Adversarial Attacks for Vision-Language Modelscited by 0
  22. 2026Overloading Large Vision-Language Models for Jailbreakingcited by 0
  23. 2026One Attack to Fool Them All: Highly Transferable Black-Box Adversarial Attacks on Frontier MLLMscited by 0
  24. 2026Which Image Property Carries the Jailbreak? A Controlled Dissection of Image-to-Text Jailbreakscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025PolyJailbreak: Cross-Modal Jailbreaking Attacks on Black-Box Multimodal LLMs
  2. 2025Jailbreaking Multimodal Large Language Models via Shuffle Inconsistency
  3. 2025VRSA: Jailbreaking Multimodal Large Language Models through Visual Reasoning Sequential Attack
  4. 2025Align is not Enough: Multimodal Universal Jailbreak Attack against Multimodal Large Language Models
  5. 2024Cross-modality Information Check for Detecting Jailbreaking in Multimodal Large Language Models
  6. 2026COMIC: Reference-Aware Safety Gating for Multimodal Large Language Models
  7. 2025JPS: Jailbreak Multimodal Large Language Models with Collaborative Visual Perturbation and Textual Steering
  8. 2024Divide and Conquer: A Hybrid Strategy Defeats Multimodal Large Language Models