# Visual Contextual Attack: Jailbreaking MLLMs with Image-Driven Context Injection

arXiv: https://arxiv.org/abs/2507.02844 · HTML: https://mlsanity.com/paper/2507.02844/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 24 of its 49 references that are in the map.
- First posted on arXiv 3 Jul 2025.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM jailbreaks and adversarial robustness](https://mlsanity.com/mllm/mllm-jailbreaks-and-adversarial-robustness/)
- LLM safety and security / [Vision-language and text-to-image safety](https://mlsanity.com/safety/vision-language-and-text-to-image-safety/)

## Builds on (most cited first, 24 of 24)

- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://mlsanity.com/paper/2310.03693/) (2023)
- [Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations](https://mlsanity.com/paper/2310.06387/) (2023)
- [FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts](https://mlsanity.com/paper/2311.05608/) (2023)
- [Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack](https://mlsanity.com/paper/2404.01833/) (2024)
- [SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models](https://mlsanity.com/paper/2402.05044/) (2024)
- [X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agents](https://mlsanity.com/paper/2504.13203/) (2025)
- [MLLM-Protector: Ensuring MLLM's Safety without Hurting Performance](https://mlsanity.com/paper/2401.02906/) (2024)
- [DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedback](https://mlsanity.com/paper/2311.10081/) (2023)
- [H-CoT: Hijacking the Chain-of-Thought Safety Reasoning Mechanism to Jailbreak Large Reasoning Models, Including OpenAI o1/o3, DeepSeek-R1, and Gemini 2.0 Flash Thinking](https://mlsanity.com/paper/2502.12893/) (2025)
- [Chain of Attack: a Semantic-Driven Contextual Multi-Turn attacker for LLM](https://mlsanity.com/paper/2405.05610/) (2024)
- [JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Models](https://mlsanity.com/paper/2407.01599/) (2024)
- [A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluations](https://mlsanity.com/paper/2502.14881/) (2025)
- [Speak Out of Turn: Safety Vulnerability of Large Language Models in Multi-turn Dialogue](https://mlsanity.com/paper/2402.17262/) (2024)
- [Safety in Large Reasoning Models: A Survey](https://mlsanity.com/paper/2504.17704/) (2025)
- [VLSBench: Unveiling Visual Leakage in Multimodal Safety](https://mlsanity.com/paper/2411.19939/) (2024)
- [ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time](https://mlsanity.com/paper/2410.06625/) (2024)
- [Jailbreaking Multimodal Large Language Models via Shuffle Inconsistency](https://mlsanity.com/paper/2501.04931/) (2025)
- [Bypassing the Safety Training of Open-Source LLMs with Priming Attacks](https://mlsanity.com/paper/2312.12321/) (2023)
- [T2ISafety: Benchmark for Assessing Fairness, Toxicity, and Privacy in Image Generation](https://mlsanity.com/paper/2501.12612/) (2025)
- [The Better Angels of Machine Personality: How Personality Relates to LLM Safety](https://mlsanity.com/paper/2407.12344/) (2024)
- [From GPT-4 to Gemini and Beyond: Assessing the Landscape of MLLMs on Generalizability, Trustworthiness and Causality through Four Modalities](https://mlsanity.com/paper/2401.15071/) (2024)
- [Dialogue Injection Attack: Jailbreaking LLMs through Context Manipulation](https://mlsanity.com/paper/2503.08195/) (2025)

## Built on by (most cited first, 25 of 25)

- [Layer-Aware Representation Filtering: Purifying Finetuning Data to Preserve LLM Safety Alignment](https://mlsanity.com/paper/2507.18631/) (2025)
- [OmniSafeBench-MM: A Unified Benchmark and Toolbox for Multimodal Jailbreak Attack-Defense Evaluation](https://mlsanity.com/paper/2512.06589/) (2025)
- [Response Attack: Exploiting Contextual Priming to Jailbreak Large Language Models](https://mlsanity.com/paper/2507.05248/) (2025)
- [SAVeS: Steering Safety Judgments in Vision-Language Models via Semantic Cues](https://mlsanity.com/paper/2603.19092/) (2026)
- [Visual Exclusivity Attacks: Automatic Multimodal Red Teaming via Agentic Planning](https://mlsanity.com/paper/2603.20198/) (2026)
- [Adversarial Attacks on Multimodal Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2603.27918/) (2026)
- [TreeTeaming: Autonomous Red-Teaming of Vision-Language Models via Hierarchical Strategy Exploration](https://mlsanity.com/paper/2603.22882/) (2026)
- [Penny Wise, Pixel Foolish: Bypassing Price Constraints in Multimodal Agents via Visual Adversarial Perturbations](https://mlsanity.com/paper/2604.16515/) (2026)
- [Empowering Multimodal LLMs with External Tools: A Comprehensive Survey](https://mlsanity.com/paper/2508.10955/) (2025)
- [Self-adaptive Dataset Construction for Real-World Multimodal Safety Scenarios](https://mlsanity.com/paper/2509.04403/) (2025)
- [Red-teaming the Multimodal Reasoning: Jailbreaking Vision-Language Models via Cross-modal Entanglement Attacks](https://mlsanity.com/paper/2602.10148/) (2026)
- [Visual Memory Injection Attacks for Multi-Turn Conversations](https://mlsanity.com/paper/2602.15927/) (2026)
- [Every Picture Tells a Dangerous Story: Memory-Augmented Multi-Agent Jailbreak Attacks on VLMs](https://mlsanity.com/paper/2604.12616/) (2026)
- [LLaVAShield: Safeguarding Multimodal Multi-Turn Dialogues in Vision-Language Models](https://mlsanity.com/paper/2509.25896/) (2025)
- [The Plot Twist: Jailbreaking Unified Multimodal Models with a Three-Act NarrativeAttack](https://mlsanity.com/paper/2509.26473/) (2025)
- [GAMBIT: A Gamified Jailbreak Framework for Multimodal Large Language Models](https://mlsanity.com/paper/2601.03416/) (2026)
- [The Salami Slicing Threat: Exploiting Cumulative Risks in LLM Systems](https://mlsanity.com/paper/2604.11309/) (2026)
- [Dictionary-Aligned Concept Control for Safeguarding Multimodal LLMs](https://mlsanity.com/paper/2604.08846/) (2026)
- [Seeing No Evil: Blinding Large Vision-Language Models to Safety Instructions via Adversarial Attention Hijacking](https://mlsanity.com/paper/2604.10299/) (2026)
- [New Wide-Net-Casting Jailbreak Attacks Risk Large Models](https://mlsanity.com/paper/2605.17128/) (2026)
- [Modeling Complex Behaviors: Multi-Personality Composition and Dynamic Switching in Vision-Language Models](https://mlsanity.com/paper/2606.11074/) (2026)
- [Security and Privacy in Retrieval-Augmented Generation: Architectures, Threats, Defenses, and Future Directions for Building Trustworthy Systems](https://mlsanity.com/paper/2606.25533/) (2026)
- [Fully Unleashing the Multimodal Attacker: Meta-Adaptive Jailbreaking of Vision-Language Models](https://mlsanity.com/paper/2608.27531/) (2026)
- [Which Image Property Carries the Jailbreak? A Controlled Dissection of Image-to-Text Jailbreaks](https://mlsanity.com/paper/2610.07009/) (2026)
- [Scenes as an Attack Surface: Uncovering Scene-Grounded Safety Failures in MLLMs](https://mlsanity.com/paper/2610.14094/) (2026)

## Abstract (arXiv)

With the emergence of strong vision language capabilities, multimodal large language models (MLLMs) have demonstrated tremendous potential for real-world applications. However, the security vulnerabilities exhibited by the visual modality pose significant challenges to deploying such models in open-world environments. Recent studies have successfully induced harmful responses from target MLLMs by encoding harmful textual semantics directly into visual inputs. However, in these approaches, the visual modality primarily serves as a trigger for unsafe behavior, often exhibiting semantic ambiguity and lacking grounding in realistic scenarios. In this work, we define a novel setting: vision-centric jailbreak, where visual information serves as a necessary component in constructing a complete and realistic jailbreak context. Building on this setting, we propose the VisCo (Visual Contextual) Attack. VisCo fabricates contextual dialogue using four distinct vision-focused strategies, dynamically generating auxiliary images when necessary to construct a vision-centric jailbreak scenario. To maximize attack effectiveness, it incorporates automatic toxicity obfuscation and semantic refinement to produce a final attack prompt that reliably triggers harmful responses from the target black-box MLLMs. Specifically, VisCo achieves a toxicity score of 4.78 and an Attack Success Rate (ASR) of 85% on MM-SafetyBench against GPT-4o, significantly outperforming the baseline, which achieves a toxicity score of 2.48 and an ASR of 22.2%. Code: https://github.com/Dtc7w3PQ/Visco-Attack.