mlsanityData to 14 Oct 2026

XSPA: Crafting Imperceptible X-Shaped Sparse Adversarial Perturbations for Transferable Attacks on VLMs

arXiv 2603.28568 ↗First posted 30 Mar 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Vision-language models (VLMs) share visual-textual representations across zero-shot classification, image captioning, and visual question answering (VQA), creating a pathway through which subtle perturbations may cause failures across tasks. We introduce X-shaped Sparse Pixel Attack (XSPA), a structured attack that restricts perturbations to two intersecting diagonal lines. Within this fixed support, XSPA jointly optimizes a classification objective, cross-task semantic guidance, perturbation magnitude, and linewise smoothness. Under the default setting, it changes about 1.04% of image pixels. On COCO, XSPA reduces zero-shot accuracy by 52.33 points on OpenAI CLIP ViT-L/14 and 67.00 points on OpenCLIP ViT-B/16; GPT-4-based caption consistency and VQA correctness decrease by up to 58.60 and 44.25 points, respectively. Matched-budget experiments show that an adaptive saliency Top-k support achieves higher attack success than the fixed X-shaped support, while XSPA provides better SSIM and LPIPS but worse PSNR and MAE. Same-image analysis finds no three-task targeted success, indicating that strong surrogate classification effects do not reliably translate into targeted output-space failures. These results position XSPA as a controlled stress test for studying how sparse fixed geometry affects VLM robustness, rather than as a universally superior attack.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 14 of its 29 references that are in the map. First posted on arXiv 30 Mar 2026.

Builds on

14 of 29 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025R-TPT: Improving Adversarial Robustness of Vision-Language Models through Test-Time Prompt Tuning15 Apr
  2. 2024Chain of Attack: On the Robustness of Vision-Language Models Against Transfer-Based Adversarial Attacks24 Nov
  3. 2022Towards Adversarial Attack on Vision-Language Pre-training Models19 Jun

Other references · 11

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  4. 2015DeepFool: a simple and accurate method to fool deep neural networkscited by 1,318
  5. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  6. 2017One pixel attack for fooling deep neural networkscited by 582
  7. 2017Adversarial Patchcited by 517
  8. 2022Shadows can be Dangerous: Stealthy and Effective Physical-world Adversarial Attack by Natural Phenomenoncited by 101
  9. 2018SparseFool: a few pixels make a big differencecited by 94
  10. 2020SPAA: Stealthy Projector-based Adversarial Attacks on Deep Image Classifierscited by 17
  11. 2024Sparse-PGD: A Unified Framework for Sparse Adversarial Perturbations Generationcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Transferable Adversarial Attacks on Black-Box Vision-Language Models
  2. 2026Unveiling the Fragility of Vision-Language Models: Multi-Modal Adversarial Synergy via Texture-Constrained Perturbations and Cross-Modal Optimization
  3. 2025Revisiting CroPA: A Reproducibility Study and Enhancements for Cross-Prompt Adversarial Transferability in Vision-Language Models
  4. 2026PDA: Text-Augmented Defense Framework for Robust Vision-Language Models against Adversarial Image Attacks
  5. 2024Enhancing Cross-Prompt Transferability in Vision-Language Models through Contextual Injection of Target Tokens
  6. 2026On Success and Simplicity: A Second Look at Transferable Vision-Language Attack Pipeline
  7. 2025Attention-Guided Patch-Wise Sparse Adversarial Attacks on Vision-Language-Action Models
  8. 2024Probing the Robustness of Vision-Language Pretrained Models: A Multimodal Adversarial Attack Approach