# Ask, Attend, Attack: A Effective Decision-Based Black-Box Targeted Attack for Image-to-Text Models

arXiv: https://arxiv.org/abs/2408.08989 · HTML: https://mlsanity.com/paper/2408.08989/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 52 references that are in the map.
- First posted on arXiv 16 Aug 2024.

## Builds on (most cited first, 24 of 24)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [VGGFace2: A dataset for recognising faces across pose and age](https://mlsanity.com/paper/1710.08092/) (2017)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [On Evaluating Adversarial Robustness of Large Vision-Language Models](https://mlsanity.com/paper/2305.16934/) (2023)
- [Efficient Decision-based Black-box Adversarial Attacks on Face Recognition](https://mlsanity.com/paper/1904.04433/) (2019)
- [Dual Attention Suppression Attack: Generate Adversarial Camouflage in Physical World](https://mlsanity.com/paper/2103.01050/) (2021)
- [Unrestricted Adversarial Examples via Semantic Manipulation](https://mlsanity.com/paper/1904.06347/) (2019)
- [Person Re-identification Method Based on Color Attack and Joint Defence](https://mlsanity.com/paper/2111.09571/) (2021)
- [Cross-Modality Perturbation Synergy Attack for Person Re-identification](https://mlsanity.com/paper/2401.10090/) (2024)
- [IoU Attack: Towards Temporally Coherent Black-Box Adversarial Attack for Visual Object Tracking](https://mlsanity.com/paper/2103.14938/) (2021)
- [Efficient Decision-based Black-box Patch Attacks on Video Recognition](https://mlsanity.com/paper/2303.11917/) (2023)
- [Beyond Dropout: Robust Convolutional Neural Networks Based on Local Feature Masking](https://mlsanity.com/paper/2407.13646/) (2024)
- [I See Dead People: Gray-Box Adversarial Attack on Image-To-Text Models](https://mlsanity.com/paper/2306.07591/) (2023)
- [Mimic and Fool: A Task Agnostic Adversarial Attack](https://mlsanity.com/paper/1906.04606/) (2019)
- [Manifold Interpolation for Large-Scale Multi-Objective Optimization via Generative Adversarial Networks](https://mlsanity.com/paper/2101.02932/) (2021)
- [Multi-View Subgraph Neural Networks: Self-Supervised Learning with Scarce Labeled Data](https://mlsanity.com/paper/2404.12569/) (2024)
- [Improving Performance Insensitivity of Large-scale Multiobjective Optimization via Monte Carlo Tree Search](https://mlsanity.com/paper/2304.04071/) (2023)

## Built on by (most cited first, 5 of 5)

- [Phy124: Fast Physics-Driven 4D Content Generation from a Single Image](https://mlsanity.com/paper/2409.07179/) (2024)
- [LLM-Virus: Evolutionary Jailbreak Attack on Large Language Models](https://mlsanity.com/paper/2501.00055/) (2024)
- [Fading the Digital Ink: A Universal Black-Box Attack Framework for 3DGS Watermarking Systems](https://mlsanity.com/paper/2508.07263/) (2025)
- [Enhancing Large-scale UAV Route Planing with Global and Local Features via Reinforcement Graph Fusion](https://mlsanity.com/paper/2412.15537/) (2024)
- [Grad-CAM for Vision Transformers: A Systematic Taxonomy and Audit of Methodological Ambiguity in Explainable AI](https://mlsanity.com/paper/2608.05258/) (2026)

## Abstract (arXiv)

While image-to-text models have demonstrated significant advancements in various vision-language tasks, they remain susceptible to adversarial attacks. Existing white-box attacks on image-to-text models require access to the architecture, gradients, and parameters of the target model, resulting in low practicality. Although the recently proposed gray-box attacks have improved practicality, they suffer from semantic loss during the training process, which limits their targeted attack performance. To advance adversarial attacks of image-to-text models, this paper focuses on a challenging scenario: decision-based black-box targeted attacks where the attackers only have access to the final output text and aim to perform targeted attacks. Specifically, we formulate the decision-based black-box targeted attack as a large-scale optimization problem. To efficiently solve the optimization problem, a three-stage process textitAsk, Attend, Attack, called textitAAA, is proposed to coordinate with the solver. textitAsk guides attackers to create target texts that satisfy the specific semantics. textitAttend identifies the crucial regions of the image for attacking, thus reducing the search space for the subsequent textitAttack. textitAttack uses an evolutionary algorithm to attack the crucial regions, where the attacks are semantically related to the target texts of textitAsk, thus achieving targeted attacks without semantic loss. Experimental results on transformer-based and CNN+RNN-based image-to-text models confirmed the effectiveness of our proposed textitAAA.