mlsanityData to 14 Oct 2026

Structure-Guided Visual Perturbation Neutralization for LVLMs

arXiv 2605.27927 ↗First posted 27 May 2026cs.CV cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Image inputs enable Large Vision Language Models (LVLMs) to perceive fine-grained visual information, but also introduce a pixel-level attack surface through which adversarial perturbations can elicit unsafe model behaviors. However, most existing defenses are designed for traditional computer vision settings and thus often overlook the cross-modal alignment required by LVLMs, leading to degraded performance. Meanwhile, the limited defenses tailored to LVLMs often require substantial image modifications and introduce considerable computational overhead, thereby compromising inference quality and efficiency. To address these limitations, we propose Structure-Induced Guided Neutralization (SIGN), a lightweight, plug-and-play defense framework that improves LVLM compatibility via Prior Structural Extraction and achieves efficient perturbation suppression via Dynamic Guided Neutralization. Extensive experiments show that SIGN achieves over 87% defense success rate with only 0.5% pixel modification and 0.16 seconds per image, while nearly preserving original visual representations and benign task performance. Our work offers a lightweight alternative to defenses that require costly model training and highlights the potential of exploiting a vision encoder for efficient adversarial protection. Our code is open source on https://anonymous.4open.science/r/SIGN-BCB1.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 34 of its 55 references that are in the map. First posted on arXiv 27 May 2026.

Builds on

34 of 55 references.

Closest ancestors · 13

Same direction, newest first

  1. 2026A Provable Energy-Guided Test-Time Defense Boosting Adversarial Robustness of Large Vision-Language Models27 Mar
  2. 2026VALD: Multi-Stage Vision Attack Detection for Efficient LVLM Defense23 Feb
  3. 2025SHIELD: Classifier-Guided Prompting for Robust and Safer LVLMs15 Oct
  4. 2025JaiLIP: Jailbreaking Vision-Language Models via Loss Guided Image Perturbation24 Sep
  5. 2025Resource Consumption Red-Teaming for Large Vision-Language Models24 Jul
  6. 2025LingoLoop Attack: Trapping MLLMs via Linguistic Context and State Entrapment into Endless Loops17 Jun
  7. 2025Con Instruction: Universal Jailbreaking of Multimodal Large Language Models via Non-Textual Modalities31 May
  8. 2025AMIA: Automatic Masking and Joint Intention Analysis Makes LVLMs Robust Jailbreak Defenders30 May
  9. 2024Exploiting Vision Encoder Vulnerabilities for Universal Adversarial Perturbations on Large Vision-Language Models11 Dec
  10. 2024PIP: Detecting Adversarial Examples in Large Vision-Language Models via Attention Patterns of Irrelevant Probe Questions8 Sep
  11. 2024Jailbreak Vision Language Models via Bi-Modal Adversarial Prompt6 Jun
  12. 2023On the Robustness of Large Multimodal Models Against Image Adversarial Attacks6 Dec
1 more
  1. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts9 Nov

Other references · 21

  1. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  6. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  7. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  9. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,340
  10. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  11. 2016Universal adversarial perturbationscited by 784
  12. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
9 more
  1. 2019Adversarial Training for Free!cited by 643
  2. 2017Countering Adversarial Images using Input Transformationscited by 520
  3. 2017Adversarial Patchcited by 517
  4. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 486
  5. 2018Defense-GAN: Protecting Classifiers Against Adversarial Attacks Using Generative Modelscited by 362
  6. 2018Feature Denoising for Improving Adversarial Robustnesscited by 314
  7. 2024Energy-Latency Manipulation of Multi-modal Large Language Models via Verbose Samplescited by 14
  8. 2025DnLUT: Ultra-Efficient Color Image Denoising via Channel-Aware Lookup Tablescited by 11
  9. 2025SHIELD: Suppressing Hallucinations In LVLM Encoders via Bias and Vulnerability Defensecited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Grounding-Driven Attack: Improving Encoder-based Adversarial Transferability against Large Vision-Language Models
  2. 2025Robust Anti-Backdoor Instruction Tuning in LVLMs
  3. 2024Defending LVLMs Against Vision Attacks through Partial-Perception Supervision
  4. 2025Safeguarding Vision-Language Models: Mitigating Vulnerabilities to Gaussian Noise in Perturbation-based Attacks
  5. 2026Breaking the weakest link to evade vision language models
  6. 2026PA-Attack: Guiding Gray-Box Attacks on LVLM Vision Encoders with Prototypes and Attention
  7. 2025Multi-Faceted Attack: Exposing Cross-Model Vulnerabilities in Defense-Equipped Vision-Language Models
  8. 2026Crafting Adversarial Inputs for Large Vision-Language Models Using Black-Box Optimization