mlsanityData to 14 Oct 2026

Revisiting CroPA: A Reproducibility Study and Enhancements for Cross-Prompt Adversarial Transferability in Vision-Language Models

arXiv 2506.22982 ↗First posted 28 Jun 2025cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Large Vision-Language Models (VLMs) have revolutionized computer vision, enabling tasks such as image classification, captioning, and visual question answering. However, they remain highly vulnerable to adversarial attacks, particularly in scenarios where both visual and textual modalities can be manipulated. In this study, we conduct a comprehensive reproducibility study of "An Image is Worth 1000 Lies: Adversarial Transferability Across Prompts on Vision-Language Models" validating the Cross-Prompt Attack (CroPA) and confirming its superior cross-prompt transferability compared to existing baselines. Beyond replication we propose several key improvements: (1) A novel initialization strategy that significantly improves Attack Success Rate (ASR). (2) Investigate cross-image transferability by learning universal perturbations. (3) A novel loss function targeting vision encoder attention mechanisms to improve generalization. Our evaluation across prominent VLMs -- including Flamingo, BLIP-2, and InstructBLIP as well as extended experiments on LLaVA validates the original results and demonstrates that our improvements consistently boost adversarial effectiveness. Our work reinforces the importance of studying adversarial vulnerabilities in VLMs and provides a more robust framework for generating transferable adversarial examples, with significant implications for understanding the security of VLMs in real-world applications.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 39 of its 46 references that are in the map. First posted on arXiv 28 Jun 2025.

Builds on

39 of 46 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024Exploiting Vision Encoder Vulnerabilities for Universal Adversarial Perturbations on Large Vision-Language Models11 Dec
  2. 2024Break the Visual Perception: Adversarial Attacks Targeting Encoded Visual Tokens of Large Vision-Language Models9 Oct
  3. 2024Enhancing Cross-Prompt Transferability in Vision-Language Models through Contextual Injection of Target Tokens19 Jun
  4. 2024One Perturbation is Enough: On Generating Universal Adversarial Perturbations against Vision-Language Pre-training Models8 Jun
  5. 2024Universal Adversarial Perturbations for Vision-Language Pre-trained Models9 May
  6. 2024An Image Is Worth 1000 Lies: Adversarial Transferability across Prompts on Vision-Language Models14 Mar
  7. 2023On the Robustness of Large Multimodal Models Against Image Adversarial Attacks6 Dec
  8. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Models22 Jun
  9. 2023On Evaluating Adversarial Robustness of Large Vision-Language Models26 May
  10. 2022Towards Adversarial Attack on Vision-Language Pre-training Models19 Jun

Other references · 29

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  5. 2013Intriguing properties of neural networkscited by 5,155
  6. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  9. 2016Universal adversarial perturbationscited by 784
  10. 2018Improving Transferability of Adversarial Examples with Input Diversitycited by 599
  11. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  12. 2016Delving into Transferable Adversarial Examples and Black-box Attackscited by 536
17 more
  1. 2021Admix: Enhancing the Transferability of Adversarial Attackscited by 156
  2. 2023On the Adversarial Robustness of Multi-Modal Foundation Modelscited by 145
  3. 2023How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMscited by 108
  4. 2019Cross-Domain Transferability of Adversarial Perturbationscited by 92
  5. 2021Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Modelscited by 75
  6. 2018Gradient Descent with Random Initialization: Fast Global Convergence for Nonconvex Phase Retrievalcited by 71
  7. 2017Fast Feature Fool: A data independent approach to universal adversarial perturbationscited by 66
  8. 2020A Single-Loop Smoothed Gradient Descent-Ascent Algorithm for Nonconvex-Concave Min-Max Problemscited by 58
  9. 2021Human-Adversarial Visual Question Answeringcited by 57
  10. 2023Boosting Adversarial Transferability by Achieving Flat Local Maximacited by 48
  11. 2019Enhancing Cross-task Black-Box Transferability of Adversarial Examples with Dispersion Reductioncited by 47
  12. 2021On the Efficacy of Adversarial Data Collection for Question Answering: Results from a Large-Scale Randomized Studycited by 31
  13. 2019Exact Adversarial Attack to Image Captioning via Structured Output Learning with Latent Variablescited by 26
  14. 2018Task-generalizable Adversarial Attack based on Perceptual Metriccited by 18
  15. 2021Controlled Caption Generation for Images Through Adversarial Attackscited by 9
  16. 2022longhorns at DADC 2022: How many linguists does it take to fool a Question Answering model? A systematic approach to adversarial attackscited by 8
  17. 2025Information-Geometric Inverse Distillation for Enhancing Adversarial Transferabilitycited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026On Success and Simplicity: A Second Look at Transferable Vision-Language Attack Pipeline
  2. 2025Transferable Adversarial Attacks on Black-Box Vision-Language Models
  3. 2026Grounding-Driven Attack: Improving Encoder-based Adversarial Transferability against Large Vision-Language Models
  4. 2026Improving Adversarial Transferability on Vision-Language Pre-training Models via Surrogate-Specific Bias Correction
  5. 2026GhostPrompt: Cross-Image Adversarial Prompt for Vision-Language Models
  6. 2024Prompt-Driven Contrastive Learning for Transferable Adversarial Attacks
  7. 2023Exploring Transferability of Multimodal Adversarial Samples for Vision-Language Pre-training Models with Contrastive Learning
  8. 2023InstructTA: Instruction-Tuned Targeted Attack for Large Vision-Language Models