# VQAttack: Transferable Adversarial Attacks on Visual Question Answering via Pre-trained Models

arXiv: https://arxiv.org/abs/2402.11083 · HTML: https://mlsanity.com/paper/2402.11083/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 41 references that are in the map.
- First posted on arXiv 16 Feb 2024.

## Builds on (most cited first, 23 of 23)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment](https://mlsanity.com/paper/1907.11932/) (2019)
- [Improving Transferability of Adversarial Examples with Input Diversity](https://mlsanity.com/paper/1803.06978/) (2018)
- [BERT-ATTACK: Adversarial Attack Against BERT Using BERT](https://mlsanity.com/paper/2004.09984/) (2020)
- [VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts](https://mlsanity.com/paper/2111.02358/) (2021)
- [Gradient-based Adversarial Attacks against Text Transformers](https://mlsanity.com/paper/2104.13733/) (2021)
- [Vision-Language Pre-Training with Triple Contrastive Learning](https://mlsanity.com/paper/2202.10401/) (2022)
- [A Self-supervised Approach for Adversarial Robustness](https://mlsanity.com/paper/2006.04924/) (2020)
- [Skip Connections Matter: On the Transferability of Adversarial Examples Generated with ResNets](https://mlsanity.com/paper/2002.05990/) (2020)
- [Towards Adversarial Attack on Vision-Language Pre-training Models](https://mlsanity.com/paper/2206.09391/) (2022)
- [Enhancing Adversarial Example Transferability with an Intermediate Level Attack](https://mlsanity.com/paper/1907.10823/) (2019)
- [Contextualized Perturbation for Textual Adversarial Attack](https://mlsanity.com/paper/2009.07502/) (2020)
- [Perturbing Across the Feature Hierarchy to Improve Standard and Strict Blackbox Attack Transferability](https://mlsanity.com/paper/2004.14861/) (2020)
- [FDA: Feature Disruptive Attack](https://mlsanity.com/paper/1909.04385/) (2019)
- [Transferable Perturbations of Deep Feature Distributions](https://mlsanity.com/paper/2004.12519/) (2020)
- [Enhancing Cross-task Black-Box Transferability of Adversarial Examples with Dispersion Reduction](https://mlsanity.com/paper/1911.11616/) (2019)
- [R&R: Metric-guided Adversarial Sentence Generation](https://mlsanity.com/paper/2104.08453/) (2021)

## Built on by (most cited first, 10 of 10)

- [Transfer Attack for Bad and Good: Explain and Boost Adversarial Transferability across Multimodal Large Language Models](https://mlsanity.com/paper/2405.20090/) (2024)
- [HV-Attack: Hierarchical Visual Attack for Multimodal Retrieval Augmented Generation](https://mlsanity.com/paper/2511.15435/) (2025)
- [Adversarial attacks against Modern Vision-Language Models](https://mlsanity.com/paper/2603.16960/) (2026)
- [FALCON: Fine-grained Activation Manipulation by Contrastive Orthogonal Unalignment for Large Language Model](https://mlsanity.com/paper/2502.01472/) (2025)
- [See No Evil: Adversarial Attacks Against Linguistic-Visual Association in Referring Multi-Object Tracking Systems](https://mlsanity.com/paper/2509.02028/) (2025)
- [The Power of Decaying Steps: Enhancing Attack Stability and Transferability for Sign-based Optimizers](https://mlsanity.com/paper/2602.19096/) (2026)
- [SurvAttack: Black-Box Attack On Survival Models through Ontology-Informed EHR Perturbation](https://mlsanity.com/paper/2412.18706/) (2024)
- [Hierarchical Refinement of Universal Multimodal Attacks on Vision-Language Models](https://mlsanity.com/paper/2601.10313/) (2026)
- [Devling into Adversarial Transferability on Image Classification: Review, Benchmark, and Evaluation](https://mlsanity.com/paper/2602.23117/) (2026)
- [HQA-VLAttack: Towards High Quality Adversarial Attack on Vision-Language Pre-Trained Models](https://mlsanity.com/paper/2604.16499/) (2026)

## Abstract (arXiv)

Visual Question Answering (VQA) is a fundamental task in computer vision and natural language process fields. Although the ``pre-training & finetuning'' learning paradigm significantly improves the VQA performance, the adversarial robustness of such a learning paradigm has not been explored. In this paper, we delve into a new problem: using a pre-trained multimodal source model to create adversarial image-text pairs and then transferring them to attack the target VQA models. Correspondingly, we propose a novel VQAttack model, which can iteratively generate both image and text perturbations with the designed modules: the large language model (LLM)-enhanced image attack and the cross-modal joint attack module. At each iteration, the LLM-enhanced image attack module first optimizes the latent representation-based loss to generate feature-level image perturbations. Then it incorporates an LLM to further enhance the image perturbations by optimizing the designed masked answer anti-recovery loss. The cross-modal joint attack module will be triggered at a specific iteration, which updates the image and text perturbations sequentially. Notably, the text perturbation updates are based on both the learned gradients in the word embedding space and word synonym-based substitution. Experimental results on two VQA datasets with five validated models demonstrate the effectiveness of the proposed VQAttack in the transferable attack setting, compared with state-of-the-art baselines. This work reveals a significant blind spot in the ``pre-training & fine-tuning'' paradigm on VQA tasks. Source codes will be released.