mlsanityData to 14 Oct 2026

Innocence in the Crossfire: Roles of Skip Connections in Jailbreaking Visual Language Models

arXiv 2507.13761 ↗First posted 18 Jul 2025cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

Language models are highly sensitive to prompt formulations - small changes in input can drastically alter their output. This raises a critical question: To what extent can prompt sensitivity be exploited to generate inapt content? In this paper, we investigate how discrete components of prompt design influence the generation of inappropriate content in Visual Language Models (VLMs). Specifically, we analyze the impact of three key factors on successful jailbreaks: (a) the inclusion of detailed visual information, (b) the presence of adversarial examples, and (c) the use of positively framed beginning phrases. Our findings reveal that while a VLM can reliably distinguish between benign and harmful inputs in unimodal settings (text-only or image-only), this ability significantly degrades in multimodal contexts. Each of the three factors is independently capable of triggering a jailbreak, and we show that even a small number of in-context examples (as few as three) can push the model toward generating inappropriate outputs. Furthermore, we propose a framework that utilizes a skip-connection between two internal layers of the VLM, which substantially increases jailbreak success rates, even when using benign images. Finally, we demonstrate that memes, often perceived as humorous or harmless, can be as effective as toxic visuals in eliciting harmful content, underscoring the subtle and complex vulnerabilities of VLMs.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 46 of its 56 references that are in the map. First posted on arXiv 18 Jul 2025.

Builds on

46 of 56 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models26 Jul

Other references · 45

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2016Layer Normalizationcited by 5,669
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  8. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  9. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  10. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  11. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  12. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
33 more
  1. 2021Transformers in Vision: A Surveycited by 1,013
  2. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  3. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  4. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  5. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  6. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  7. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 490
  8. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 431
  9. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreakercited by 367
  10. 2023Are aligned neural networks adversarially aligned?cited by 315
  11. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 297
  12. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easilycited by 259
  13. 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70Bcited by 195
  14. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 191
  15. 2022A Survey of Vision-Language Pre-Trained Modelscited by 177
  16. 2024ProSA: Assessing and Understanding the Prompt Sensitivity of LLMscited by 149
  17. 2023AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Modelscited by 111
  18. 2025Words or Vision: Do Vision-Language Models Have Blind Faith in Text?cited by 89
  19. 2023VLATTACK: Multimodal Adversarial Attacks on Vision-Language Tasks via Pre-trained Modelscited by 68
  20. 2023Adversarial Demonstration Attacks on Large Language Modelscited by 63
  21. 2021Model Extraction and Adversarial Transferability, Your BERT is Vulnerable!cited by 53
  22. 2024How Susceptible are LLMs to Influence in Prompts?cited by 33
  23. 2021Using Adversarial Attacks to Reveal the Statistical Bias in Machine Reading Comprehension Modelscited by 29
  24. 2023Make Them Spill the Beans! Coercive Knowledge Extraction from (Production) LLMscited by 27
  25. 2024Recent Advances in Hate Speech Moderation: Multimodality and the Role of Large Modelscited by 26
  26. 2023Analyzing the Inherent Response Tendency of LLMs: Real-World Instructions-Driven Jailbreakcited by 23
  27. 2023On Copyright Risks of Text-to-Image Diffusion Modelscited by 20
  28. 2022Single-Stream Multi-Level Alignment for Vision-Language Pretrainingcited by 16
  29. 2024ExpLLM: Towards Chain of Thought for Facial Expression Recognitioncited by 15
  30. 2024Can LLMs' Tuning Methods Work in Medical Multimodal Domain?cited by 12
  31. 2020RUArt: A Novel Text-Centered Solution for Text-Based Visual Question Answeringcited by 8
  32. 2024SAFE-MEME: Structured Reasoning Framework for Robust Hate Speech Detection in Memescited by 4
  33. 2023Expanding Scope: Adapting English Adversarial Attacks to Chinesecited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Jailbreaks on Vision Language Model via Multimodal Reasoning
  2. 2026Adversarial Style Optimization: Enhancing VLM Jailbreaks by GRPO-based Stylistic Triggers Optimization
  3. 2024Jailbreak Vision Language Models via Bi-Modal Adversarial Prompt
  4. 2026Low-Effort Jailbreak Attacks Against Text-to-Image Safety Filters
  5. 2025Behind the Mask: Benchmarking Camouflaged Jailbreaks in Large Language Models
  6. 2025Anyone Can Jailbreak: Prompt-Based Attacks on LLMs and T2Is
  7. 2026Same Model, Different Weakness: How Language and Modality Reshape the Jailbreak Attack Surface in Frontier MLLMs
  8. 2025Towards Effective MLLM Jailbreaking Through Balanced On-Topicness and OOD-Intensity