mlsanityData to 7 Oct 2026

The Great Contradiction Showdown: How Jailbreak and Stealth Wrestle in Vision-Language Models?

arXiv 2410.01438 ↗First posted 2 Oct 2024cs.LGcited by 2

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

Vision-Language Models (VLMs) have achieved remarkable performance on a variety of tasks, yet they remain vulnerable to jailbreak attacks that compromise safety and reliability. In this paper, we provide an information-theoretic framework for understanding the fundamental trade-off between the effectiveness of these attacks and their stealthiness. Drawing on Fano's inequality, we demonstrate how an attacker's success probability is intrinsically linked to the stealthiness of generated prompts. Building on this, we propose an efficient algorithm for detecting non-stealthy jailbreak attacks, offering significant improvements in model robustness. Experimental results highlight the tension between strong attacks and their detectability, providing insights into both adversarial strategies and defense mechanisms.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 21 of its 70 references that are in the map. First posted on arXiv 2 Oct 2024.

Builds on

21 of 70 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Jailbreak Vision Language Models via Bi-Modal Adversarial Prompt6 Jun
  2. 2024JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks3 Apr
  3. 2023SA-Attack: Improving Adversarial Transferability of Vision-Language Pre-training Models via Self-Augmentation8 Dec
  4. 2023OT-Attack: Enhancing Adversarial Transferability of Vision-Language Models via Optimal Transport Optimization7 Dec
  5. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models29 Nov
  6. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts9 Nov

Other references · 15

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  6. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  7. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  8. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 738
  9. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  10. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  11. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  12. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 434
3 more
  1. 2023Low-Resource Languages Jailbreak GPT-4cited by 290
  2. 2017Detecting Adversarial Samples from Artifactscited by 244
  3. 2023Bergeron: Combating Adversarial Attacks through a Conscience-Based Alignment Frameworkcited by 29

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2024Jailbreak Attacks and Defenses against Multimodal Generative Models: A Surveycited by 31
  2. 2025Odysseus: Jailbreaking Commercial Multimodal LLM-integrated Systems via Dual Steganographycited by 6

Similar, not cited

Close in wording, no citation either way

  1. 2025HiddenDetect: Detecting Jailbreak Attacks against Large Vision-Language Models via Monitoring Hidden States
  2. 2025Defense-to-Attack: Bypassing Weak Defenses Enables Stronger Jailbreaks in Vision-Language Models
  3. 2025How Jailbreak Defenses Work and Ensemble? A Mechanistic Investigation
  4. 2025Jailbreak Mimicry: Automated Discovery of Narrative-Based Jailbreaks for Large Language Models
  5. 2026Jailbreaks on Vision Language Model via Multimodal Reasoning
  6. 2024ImgTrojan: Jailbreaking Vision-Language Models with ONE Image
  7. 2025Towards Effective MLLM Jailbreaking Through Balanced On-Topicness and OOD-Intensity
  8. 2026Jailbreaking Leaves a Trace: Understanding and Detecting Jailbreak Attacks from Internal Representations of Large Language Models