mlsanityData to 7 Oct 2026

Revealing Physical-World Semantic Vulnerabilities: Universal Adversarial Patch for Infrared Vision-Language Models

arXiv 2604.03117 ↗First posted 3 Apr 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Infrared vision-language models (IR-VLMs) are becoming important for semantic perception in low-visibility environments, yet their robustness to physical semantic attacks remains underexplored. Existing adversarial patch methods are mainly designed for red-green-blue (RGB) images or closed-set infrared detectors and do not directly address open-ended IR-VLM tasks, where one deployable artifact can affect classification, captioning, and visual question answering (VQA) simultaneously. We propose Universal Curved-Grid Patch, abbreviated UCGP, a universal physical adversarial patch framework tailored to IR-VLMs. UCGP represents the patch as a deployable low-frequency curved grid and optimizes it with a representation-driven objective over subspace departure, topology disruption, and local appearance regularization. Meta Differential Evolution (MetaDE) searches for patch parameters with two physical robustness augmentations: Expectation over Transformation (EOT) capture sampling for imaging variations and thin-plate spline (TPS) patch-deformation modeling for non-rigid local shape changes. Rather than manipulating labels or prompts, UCGP disrupts the clean-category manifold in visual representation space, which later appears as degraded cross-modal outputs. Experiments show that a single shared patch degrades classification, captioning, and VQA across diverse IR-VLM architectures while retaining measurable cross-model transfer, cross-dataset generalization, cross-category extensibility, and real-scene physical effectiveness under the evaluated conditions. These results reveal a robustness blind spot in current infrared multimodal systems. Code and reproducibility assets are available at https://github.com/dyx6663/UCGP.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 34 of its 49 references that are in the map. First posted on arXiv 3 Apr 2026.

Builds on

34 of 49 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025X-Transfer Attacks: Towards Super Transferable Adversarial Attacks on CLIP8 May
  2. 2025When Lighting Deceives: Exposing Vision-Language Models' Illumination Vulnerability Through Illumination Transformation Attack10 Mar
  3. 2024Chain of Attack: On the Robustness of Vision-Language Models Against Transfer-Based Adversarial Attacks24 Nov
  4. 2024AnyAttack: Towards Large-scale Self-supervised Adversarial Attacks on Vision-language Models7 Oct
  5. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Models22 Jun

Other references · 29

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  9. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  10. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  11. 2019Square Attack: a query-efficient black-box adversarial attack via random searchcited by 682
  12. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
17 more
  1. 2017Adversarial Patchcited by 514
  2. 2018Black-box Adversarial Attacks with Limited Queries and Informationcited by 499
  3. 2022Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detectioncited by 313
  4. 2023Demystifying CLIP Datacited by 265
  5. 2021LLVIP: A Visible-infrared Paired Dataset for Low-light Visioncited by 229
  6. 2018ShapeShifter: Robust Physical Adversarial Attack on Faster R-CNN Object Detectorcited by 159
  7. 2023VLATTACK: Multimodal Adversarial Attacks on Vision-Language Tasks via Pre-trained Modelscited by 66
  8. 2022Infrared Invisible Clothing:Hiding from Infrared Detectors at Multiple Angles in Real Worldcited by 42
  9. 2023Physically Adversarial Infrared Patches with Learnable Shapes and Locationscited by 38
  10. 2023Unified Adversarial Patch for Cross-modal Attacks in the Physical Worldcited by 27
  11. 2020LSOTB-TIR:A Large-Scale High-Diversity Thermal Infrared Object Tracking Benchmarkcited by 25
  12. 2023Adversarial Infrared Blocks: A Multi-view Black-box Attack to Thermal Infrared Detectors in Physical Worldcited by 11
  13. 2025IRGPT: Understanding Real-world Infrared Image with Bi-cross-modal Curriculum on Large-scale Benchmarkcited by 11
  14. 2023Adversarial Infrared Curves: An Attack on Infrared Pedestrian Detectors in the Physical Worldcited by 10
  15. 2025MetaDE: Evolving Differential Evolution by Differential Evolutioncited by 9
  16. 2025IF-Bench: Benchmarking and Enhancing MLLMs for Infrared Images with Generative Visual Promptingcited by 6
  17. 2025RGB-Th-Bench: A Dense benchmark for Visual-Thermal Understanding of Vision Language Modelscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026InfraPatch: Cross-Task Targeted Grayscale Patch Attacks on Infrared-Adapted Vision-Language Models
  2. 2026InfraQR: Edge-Placed QR-Inspired Structured Patch Attacks on Infrared Vision-Language Models
  3. 2026Exposing Vulnerabilities in Visible-Infrared VLMs: A Unified Geometric Adversarial Framework with Cross-Task Transferability
  4. 2025When Robots Obey the Patch: Universal Transferable Patch Attacks on Vision-Language-Action Models
  5. 2026AirflowAttack: Thermal-Airflow Adversarial Perturbations against Infrared Remote-Sensing Vision-Language Models
  6. 2024Safeguarding Vision-Language Models Against Patched Visual Prompt Injectors
  7. 2025Attention-Guided Patch-Wise Sparse Adversarial Attacks on Vision-Language-Action Models
  8. 2024Towards Adversarially Robust Vision-Language Models: Insights from Design Choices and Prompt Formatting Techniques