mlsanityData to 14 Oct 2026

InfraPatch: Cross-Task Targeted Grayscale Patch Attacks on Infrared-Adapted Vision-Language Models

arXiv 2609.02233 ↗First posted 2 Sep 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Infrared vision-language models (IR-VLMs) have emerged as a promising paradigm for multimodal perception under low-visibility conditions, yet their robustness to targeted adversarial attacks remains poorly understood. Existing adversarial patch methods mainly study RGB-based models or a single downstream task and do not characterize whether localized perturbations can induce an intended semantic target in IR-VLMs. We propose InfraPatch, a white-box, per-instance framework for targeted digital grayscale patch attacks against IR-VLMs. InfraPatch optimizes a compact single-channel patch within an approximately 5% local-area budget, combines proxy-guided placement with task-adaptive semantic objectives, and induces target behaviors in image classification, image captioning, and binary visual question answering. We evaluate ten infrared-adapted model variants on 300 synthetic infrared-style images generated by applying DiffV2IR to a fixed 30-category COCO subset, using clean-conditioned targeted success criteria. InfraPatch achieves targeted attack success rates from 86.00% to 100% across the ten variants. On CLIP and BLIP-2, proxy location search improves success by 6.67 and 10.33 percentage points over optimized random placement, respectively; LLaVA-1.5 remains saturated near 100% under both settings. Patch-area and objective ablations further expose substantial differences in vulnerability across architectures and task formats. These results show that small grayscale patches can inject chosen target semantics across IR-VLM families under a controlled digital threat model, motivating stronger robustness evaluation for infrared multimodal systems.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 23 of its 38 references that are in the map. First posted on arXiv 2 Sep 2026.

Builds on

23 of 38 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025When Lighting Deceives: Exposing Vision-Language Models' Illumination Vulnerability Through Illumination Transformation Attack10 Mar
  2. 2024Multimodal Adversarial Defense for Vision-Language Models by Leveraging One-To-Many Relationships29 May
  3. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts9 Nov
  4. 2023On Evaluating Adversarial Robustness of Large Vision-Language Models26 May

Other references · 19

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  6. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  8. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  9. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  10. 2017Adversarial Patchcited by 517
  11. 2019Fooling automated surveillance cameras: adversarial patches to attack person detectioncited by 268
  12. 2023Image Hijacks: Adversarial Images can Control Generative Models at Runtimecited by 172
7 more
  1. 2017Synthesizing Robust Adversarial Examplescited by 39
  2. 2023Physically Adversarial Infrared Patches with Learnable Shapes and Locationscited by 39
  3. 2023Unified Adversarial Patch for Cross-modal Attacks in the Physical Worldcited by 27
  4. 2025DiffV2IR: Visible-to-Infrared Diffusion Model via Vision-Language Understandingcited by 16
  5. 2023Adversarial Infrared Curves: An Attack on Infrared Pedestrian Detectors in the Physical Worldcited by 10
  6. 2024Omniview-Tuning: Boosting Viewpoint Invariance of Vision-Language Pre-training Modelscited by 9
  7. 2026Knowledge-Guided Adversarial Training for Infrared Object Detection via Thermal Radiation Modelingcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026InfraQR: Edge-Placed QR-Inspired Structured Patch Attacks on Infrared Vision-Language Models
  2. 2026Revealing Physical-World Semantic Vulnerabilities: Universal Adversarial Patch for Infrared Vision-Language Models
  3. 2025Attention-Guided Patch-Wise Sparse Adversarial Attacks on Vision-Language-Action Models
  4. 2026Unveiling the Fragility of Vision-Language Models: Multi-Modal Adversarial Synergy via Texture-Constrained Perturbations and Cross-Modal Optimization
  5. 2024Safeguarding Vision-Language Models Against Patched Visual Prompt Injectors
  6. 2024Patch is Enough: Naturalistic Adversarial Patch against Vision-Language Pre-training Models
  7. 2026Breaking the weakest link to evade vision language models
  8. 2024Probing the Robustness of Vision-Language Pretrained Models: A Multimodal Adversarial Attack Approach