mlsanityData to 7 Oct 2026

Seeing No Evil: Blinding Large Vision-Language Models to Safety Instructions via Adversarial Attention Hijacking

arXiv 2604.10299 ↗First posted 11 Apr 2026cs.CV cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

Large Vision-Language Models (LVLMs) rely on attention-based retrieval of safety instructions to maintain alignment during generation. Existing attacks typically optimize image perturbations to maximize harmful output likelihood, but suffer from slow convergence due to gradient conflict between adversarial objectives and the model's safety-retrieval mechanism. We propose Attention-Guided Visual Jailbreaking, which circumvents rather than overpowers safety alignment by directly manipulating attention patterns. Our method introduces two simple auxiliary objectives: (1) suppressing attention to alignment-relevant prefix tokens and (2) anchoring generation on adversarial image features. This simple yet effective push-pull formulation reduces gradient conflict by 45% and achieves 94.4% attack success rate on Qwen-VL (vs. 68.8% baseline) with 40% fewer iterations. At tighter perturbation budgets (ε=8/255), we maintain 59.0% ASR compared to 45.7% for standard methods. Mechanistic analysis reveals a failure mode we term safety blindness: successful attacks suppress system-prompt attention by 80%, causing models to generate harmful content not by overriding safety rules, but by failing to retrieve them.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 23 of its 35 references that are in the map. First posted on arXiv 11 Apr 2026.

Builds on

23 of 35 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025V-Attack: Targeting Disentangled Value Features for Controllable Adversarial Attacks on LVLMs25 Nov
  2. 2025Visual Contextual Attack: Jailbreaking MLLMs with Image-Driven Context Injection3 Jul
  3. 2025Attention! Your Vision Language Model Could Be Maliciously Manipulated26 May
  4. 2025Making Every Step Effective: Jailbreaking Large Vision-Language Models Through Hierarchical KV Equalization14 Mar
  5. 2025Distraction is All You Need for Multimodal Large Language Model Jailbreaking15 Feb
  6. 2024Exploiting Vision Encoder Vulnerabilities for Universal Adversarial Perturbations on Large Vision-Language Models11 Dec
  7. 2024PIP: Detecting Adversarial Examples in Large Vision-Language Models via Attention Patterns of Irrelevant Probe Questions8 Sep
  8. 2024Jailbreak Vision Language Models via Bi-Modal Adversarial Prompt6 Jun
  9. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models26 Jul

Other references · 14

  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  3. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  4. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  5. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  6. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  7. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  8. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  9. 2023LIMA: Less Is More for Alignmentcited by 1,080
  10. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  11. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  12. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
2 more
  1. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 268
  2. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 251

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Spot Risks Before Speaking! Unraveling Safety Attention Heads in Large Vision-Language Models
  2. 2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Models
  3. 2026Attention Is Where You Attack
  4. 2025DAVSP: Safety Alignment for Large Vision-Language Models via Deep Aligned Visual Safety Prompt
  5. 2025JailBound: Jailbreaking Internal Safety Boundaries of Vision-Language Models
  6. 2025Safeguarding Vision-Language Models: Mitigating Vulnerabilities to Gaussian Noise in Perturbation-based Attacks
  7. 2026Attention Hijacking: Response Manipulation Across Queries in Vision-Language Models
  8. 2025HiddenDetect: Detecting Jailbreak Attacks against Large Vision-Language Models via Monitoring Hidden States