mlsanityData to 14 Oct 2026

Exposing Vulnerabilities in Visible-Infrared VLMs: A Unified Geometric Adversarial Framework with Cross-Task Transferability

arXiv 2605.22273 ↗First posted 21 May 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Vision-language models (VLMs) have achieved strong performance across diverse multimodal tasks, but their adversarial robustness in visible-infrared (VIS-IR) scenarios remains underexplored. This gap is critical because VIS-IR sensing is widely used in real-world perception systems to support reliable understanding under challenging imaging conditions. To address this cross-modal threat setting, we propose CFGPatch, a curved-edge fractal geometric adversarial patch framework for attacking VIS-IR VLMs. CFGPatch builds on triangular fractal geometry and replaces rigid straight-edged primitives with Bezier-curved elements, preserving multi-scale fractal self-similarity while introducing smoother contours, richer directional variation, and more flexible shape deformation. In addition, we design a modality-specific Fraser-spiral rendering mechanism to inject fine-grained texture distortions and misleading perceptual cues into visible and infrared images. By coupling global curved-fractal geometry with local spiral-based appearance interference, CFGPatch disrupts both shape perception and texture interpretation. We further adopt expectation over transformation (EOT) to improve robustness against common image-level transformations. Extensive experiments show that CFGPatch effectively fools VIS-IR VLMs and consistently outperforms standard patch baselines in attack effectiveness and robustness. Moreover, adversarial samples optimized for zero-shot classification transfer well to image captioning and visual question answering, demonstrating strong cross-task transferability and generalizability across downstream tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 37 of its 51 references that are in the map. First posted on arXiv 21 May 2026.

Builds on

37 of 51 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025When Lighting Deceives: Exposing Vision-Language Models' Illumination Vulnerability Through Illumination Transformation Attack10 Mar
  2. 2024AdvDreamer Unveils: Are Vision-Language Models Truly Ready for Real-World 3D Variations?4 Dec
  3. 2024Chain of Attack: On the Robustness of Vision-Language Models Against Transfer-Based Adversarial Attacks24 Nov
  4. 2024Jailbreak Vision Language Models via Bi-Modal Adversarial Prompt6 Jun
  5. 2023On Evaluating Adversarial Robustness of Large Vision-Language Models26 May

Other references · 32

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  10. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  11. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  12. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
20 more
  1. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,433
  2. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  3. 2022MaPLe: Multi-modal Prompt Learningcited by 687
  4. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
  5. 2023Demystifying CLIP Datacited by 270
  6. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 243
  7. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGIcited by 193
  8. 2018ShapeShifter: Robust Physical Adversarial Attack on Faster R-CNN Object Detectorcited by 159
  9. 2018DPatch: An Adversarial Patch Attack on Object Detectorscited by 147
  10. 2022Shadows can be Dangerous: Stealthy and Effective Physical-world Adversarial Attack by Natural Phenomenoncited by 101
  11. 2022Fine-Grained Semantically Aligned Vision-Language Pre-Trainingcited by 68
  12. 2024Assessing and Learning Alignment of Unimodal Vision and Language Modelscited by 32
  13. 2023Benchmarking Robustness of Adaptation Methods on Pre-trained Vision-Language Modelscited by 31
  14. 2023Unified Adversarial Patch for Cross-modal Attacks in the Physical Worldcited by 27
  15. 2025Analysing the Robustness of Vision-Language-Models to Common Corruptionscited by 16
  16. 2024Locality Alignment Improves Vision-Language Modelscited by 13
  17. 2025IRGPT: Understanding Real-world Infrared Image with Bi-cross-modal Curriculum on Large-scale Benchmarkcited by 12
  18. 2025CDUPatch: Color-Driven Universal Adversarial Patch Attack for Dual-Modal Visible-Infrared Detectorscited by 6
  19. 2023Two-stage optimized unified adversarial patch for attacking visible-infrared cross-modal detectors in the physical worldcited by 4
  20. 2025VisMoDAl: Visual Analytics for Evaluating and Improving Corruption Robustness of Vision-Language Modelscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Revealing Physical-World Semantic Vulnerabilities: Universal Adversarial Patch for Infrared Vision-Language Models
  2. 2026InfraPatch: Cross-Task Targeted Grayscale Patch Attacks on Infrared-Adapted Vision-Language Models
  3. 2026GeoDetect: Geometric Adversarial Detection for VLPs
  4. 2026InfraQR: Edge-Placed QR-Inspired Structured Patch Attacks on Infrared Vision-Language Models
  5. 2026AirflowAttack: Thermal-Airflow Adversarial Perturbations against Infrared Remote-Sensing Vision-Language Models
  6. 2026On the Adversarial Robustness of 3D Large Vision-Language Models
  7. 2026Tex3D: Objects as Attack Surfaces via Adversarial 3D Textures for Vision-Language-Action Models
  8. 2026The Geometry of Robustness: Optimizing Loss Landscape Curvature and Feature Manifold Alignment for Robust Finetuning of Vision-Language Models