mlsanityData to 14 Oct 2026

When Surfaces Lie: Exploiting Wrinkle-Induced Attention Shift to Attack Vision-Language Models

arXiv 2603.27759 ↗First posted 29 Mar 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Visual-Language Models (VLMs) have demonstrated exceptional cross-modal understanding across various tasks, including zero-shot classification, image captioning, and visual question answering. However, their robustness to physically plausible non-rigid deformations-such as wrinkles on flexible surfaces-remains poorly understood. In this work, we propose a parametric structural perturbation method inspired by the mechanics of three-dimensional fabric wrinkles. Specifically, our method generates photorealistic non-rigid perturbations by constructing multi-scale wrinkle fields and integrating displacement field distortion with surface-consistent appearance variations. To achieve an optimal balance between visual naturalness and adversarial effectiveness, we design a hierarchical fitness function in a low-dimensional parameter space and employ an optimization-based search strategy. We evaluate our approach using a two-stage framework: perturbations are first optimized on a zero-shot classification proxy task and subsequently assessed for transferability on generative tasks. Experimental results demonstrate that our method significantly degrades the performance of various state-of-the-art VLMs, consistently outperforming baselines in both image captioning and visual question-answering tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 19 of its 33 references that are in the map. First posted on arXiv 29 Mar 2026.

Builds on

19 of 33 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025When Lighting Deceives: Exposing Vision-Language Models' Illumination Vulnerability Through Illumination Transformation Attack10 Mar
  2. 2024AdvDreamer Unveils: Are Vision-Language Models Truly Ready for Real-World 3D Variations?4 Dec
  3. 2024Adversarial Attacks of Vision Tasks in the Past 10 Years: A Survey31 Oct
  4. 2023On the Robustness of Large Multimodal Models Against Image Adversarial Attacks6 Dec
  5. 2023Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models26 Jul

Other references · 14

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  4. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  6. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  7. 2022Shadows can be Dangerous: Stealthy and Effective Physical-world Adversarial Attack by Natural Phenomenoncited by 101
  8. 2018GenAttack: Practical Black-box Attacks with Gradient-Free Optimizationcited by 82
  9. 2017Exploring the Landscape of Spatial Robustnesscited by 81
  10. 2024On the test-time zero-shot generalization of vision-language models: Do we really need prompt learning?cited by 67
  11. 2025Meta CLIP 2: A Worldwide Scaling Recipecited by 64
  12. 2017Synthesizing Robust Adversarial Examplescited by 39
2 more
  1. 2023RFLA: A Stealthy Reflected Light Adversarial Attack in the Physical Worldcited by 23
  2. 2020Metrics and methods for robustness evaluation of neural networks with generative modelscited by 12

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Anchored, Not Graded: Vision-Language Models Fail at Slant-from-Texture Perception
  2. 2026Unveiling the Fragility of Vision-Language Models: Multi-Modal Adversarial Synergy via Texture-Constrained Perturbations and Cross-Modal Optimization
  3. 2026Revealing Physical-World Semantic Vulnerabilities: Universal Adversarial Patch for Infrared Vision-Language Models
  4. 2026Style-Aware Gloss Control for Generative Non-Photorealistic Rendering
  5. 2025Beyond Static Perception: Integrating Temporal Context into VLMs for Cloth Folding
  6. 2026Understanding Pruning Regimes in Vision-Language Models Through Domain-Aware Layer Selection
  7. 2026InfraPatch: Cross-Task Targeted Grayscale Patch Attacks on Infrared-Adapted Vision-Language Models
  8. 2025On the Reliability of Vision-Language Models Under Adversarial Frequency-Domain Perturbations