mlsanityData to 14 Oct 2026

Scenes as an Attack Surface: Uncovering Scene-Grounded Safety Failures in MLLMs

arXiv 2610.14094 ↗First posted 8 Oct 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

Existing jailbreak attacks have exposed serious safety weaknesses in multimodal large language models (MLLMs). Most prior attacks rely on specially constructed visual inputs to carry or support harmful intent. In this paper, we uncover a different vulnerability: task-relevant scene semantics alone can shift an MLLM from refusal to compliance, even when the visual input introduces no additional attack instructions and the original task intent remains unchanged. We formalize this failure as a scene-grounded visual safety vulnerability and introduce SceneSafetyBench to investigate it across eight risk categories and nine white- and black-box MLLMs. Our analysis reveals that this failure is associated with how task-relevant visual evidence influences downstream safety representations, suggesting that visual grounding alone cannot fully explain the observed safety shift. Building on this finding, we propose Scene-Grounded Attacks (SGA), which preserve the original task while constructing validated scene-grounded queries and combining their independent responses. Across nine target MLLMs, SGA achieves an average attack success rate (ASR) of 86.17%, outperforming the strongest baseline by 8.55 percentage points and achieving the highest ASR on eight models. Our results identify scene semantics as an overlooked attack surface in MLLMs and highlight a growing tension between stronger visual grounding and robust multimodal safety alignment.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 39 of its 51 references that are in the map. First posted on arXiv 8 Oct 2026.

Builds on

39 of 51 references.

Closest ancestors · 17

Same direction, newest first

  1. 2026Adversarial Style Optimization: Enhancing VLM Jailbreaks by GRPO-based Stylistic Triggers Optimization1 Jun
  2. 2026DMN: A Compositional Framework for Jailbreaking Multimodal LLMs with Multi-Image Inputs18 May
  3. 2026The Side Effects of Being Smart: Safety Risks in MLLMs' Multi-Image Reasoning20 Jan
  4. 2025SafeMT: Multi-turn Safety for Multimodal Language Models14 Oct
  5. 2025JPS: Jailbreak Multimodal Large Language Models with Collaborative Visual Perturbation and Textual Steering7 Aug
  6. 2025Visual Contextual Attack: Jailbreaking MLLMs with Image-Driven Context Injection3 Jul
  7. 2025VisCRA: A Visual Chain Reasoning Attack for Jailbreaking Multimodal Large Language Models26 May
  8. 2025Jailbreaking Multimodal Large Language Models via Shuffle Inconsistency9 Jan
  9. 2024VLSBench: Unveiling Visual Leakage in Multimodal Safety29 Nov
  10. 2024IDEATOR: Jailbreaking and Benchmarking Large Vision-Language Models Using Themselves29 Oct
  11. 2024Cross-Modal Safety Mechanism Transfer in Large Vision-Language Models16 Oct
  12. 2024Multimodal Situational Safety8 Oct
5 more
  1. 2024From LLMs to MLLMs: Exploring the Landscape of Multimodal Jailbreaking21 Jun
  2. 2024JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks3 Apr
  3. 2024Images are Achilles' Heel of Alignment: Exploiting Visual Vulnerabilities for Jailbreaking Multimodal Large Language Models14 Mar
  4. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models29 Nov
  5. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts9 Nov

Other references · 22

  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  2. 2025Qwen3-VL Technical Reportcited by 2,793
  3. 2025Gemma 3 Technical Reportcited by 1,824
  4. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  5. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  6. 2025Qwen-Image Technical Reportcited by 1,044
  7. 2024ShieldGemma: Generative AI Content Moderation Based on Gemmacited by 243
  8. 2025MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipecited by 186
  9. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 137
  10. 2024SafeBench: A Safety Evaluation Framework for Multimodal Large Language Modelscited by 69
  11. 2024Visual-RolePlay: Universal Jailbreak Attack on MultiModal Large Language Models via Role-playing Image Charactercited by 68
  12. 2024Jailbreak Large Vision-Language Models Through Multi-Modal Linkagecited by 62
10 more
  1. 2025Playing the Fool: Jailbreaking LLMs and Multimodal LLMs with Out-of-Distribution Strategycited by 44
  2. 2024The Scene Language: Representing Scenes with Programs, Words, and Embeddingscited by 36
  3. 2024Heuristic-Induced Multimodal Risk Distribution Jailbreak Attack for Multimodal Large Language Modelscited by 33
  4. 2025Token Activation Map to Visually Explain Multimodal LLMscited by 32
  5. 2025FC-Attack: Jailbreaking Multimodal Large Language Models via Auto-Generated Flowchartscited by 18
  6. 2025OmniSafeBench-MM: A Unified Benchmark and Toolbox for Multimodal Jailbreak Attack-Defense Evaluationcited by 11
  7. 2025MIRAGE: Multimodal Immersive Reasoning and Guided Exploration for Red-Team Jailbreak Attackscited by 10
  8. 2026Phi-4-reasoning-vision-15B Technical Reportcited by 10
  9. 2025Benchmarking MLLM-based Web Understanding: Reasoning, Robustness and Safetycited by 6
  10. 2026Furina: Fragmented Uncertainty-Driven Refusal Instability Attackcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Spot Risks Before Speaking! Unraveling Safety Attention Heads in Large Vision-Language Models
  2. 2026COMIC: Reference-Aware Safety Gating for Multimodal Large Language Models
  3. 2026Beyond Visual Safety: Jailbreaking Multimodal Large Language Models for Harmful Image Generation via Semantic-Agnostic Inputs
  4. 2024MLLM-Protector: Ensuring MLLM's Safety without Hurting Performance
  5. 2025Security Tensors as a Cross-Modal Bridge: Extending Text-Aligned Safety to Vision in LVLM
  6. 2026MTMCS-Bench: Evaluating Contextual Safety of Multimodal Large Language Models in Multi-Turn Dialogues
  7. 2025Reimagining Safety Alignment with An Image
  8. 2026Jailbreaking Vision-Language Models Through the Visual Modality