# Exact Adversarial Attack to Image Captioning via Structured Output Learning with Latent Variables

arXiv: https://arxiv.org/abs/1905.04016 · HTML: https://mlsanity.com/paper/1905.04016/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 May 2019.

## Built on by (most cited first, 26 of 26)

- [On Evaluating Adversarial Robustness of Large Vision-Language Models](https://mlsanity.com/paper/2305.16934/) (2023)
- [Advances in adversarial attacks and defenses in computer vision: A survey](https://mlsanity.com/paper/2108.00401/) (2021)
- [Parallel Rectangle Flip Attack: A Query-based Black-box Attack against Object Detection](https://mlsanity.com/paper/2201.08970/) (2022)
- [Targeted Attack against Deep Neural Networks via Flipping Limited Weight Bits](https://mlsanity.com/paper/2102.10496/) (2021)
- [An Image Is Worth 1000 Lies: Adversarial Transferability across Prompts on Vision-Language Models](https://mlsanity.com/paper/2403.09766/) (2024)
- [Chain of Attack: On the Robustness of Vision-Language Models Against Transfer-Based Adversarial Attacks](https://mlsanity.com/paper/2411.15720/) (2024)
- [Efficient Generation of Targeted and Transferable Adversarial Examples for Vision-Language Models Via Diffusion Models](https://mlsanity.com/paper/2404.10335/) (2024)
- [Consensus Graph Representation Learning for Better Grounded Image Captioning](https://mlsanity.com/paper/2112.00974/) (2021)
- [Attacks in Adversarial Machine Learning: A Systematic Survey from the Life-cycle Perspective](https://mlsanity.com/paper/2302.09457/) (2023)
- [MirrorCheck: Efficient Adversarial Defense for Vision-Language Models](https://mlsanity.com/paper/2406.09250/) (2024)
- [MagDR: Mask-guided Detection and Reconstruction for Defending Deepfakes](https://mlsanity.com/paper/2103.14211/) (2021)
- [Fast Adversarial Training with Smooth Convergence](https://mlsanity.com/paper/2308.12857/) (2023)
- [Instant Adversarial Purification with Adversarial Consistency Distillation](https://mlsanity.com/paper/2408.17064/) (2024)
- [Versatile Weight Attack via Flipping Limited Bits](https://mlsanity.com/paper/2207.12405/) (2022)
- [Controlled Caption Generation for Images Through Adversarial Attacks](https://mlsanity.com/paper/2107.03050/) (2021)
- [Shilling Black-box Review-based Recommender Systems through Fake Review Generation](https://mlsanity.com/paper/2306.16526/) (2023)
- [Semantically Stealthy Adversarial Attacks against Segmentation Models](https://mlsanity.com/paper/2104.01732/) (2021)
- [GSE: Group-wise Sparse and Explainable Adversarial Attacks](https://mlsanity.com/paper/2311.17434/) (2023)
- [AttackGNN: Red-Teaming GNNs in Hardware Security Using Reinforcement Learning](https://mlsanity.com/paper/2402.13946/) (2024)
- [AICAttack: Adversarial Image Captioning Attack with Attention-Based Optimization](https://mlsanity.com/paper/2402.11940/) (2024)
- [Benchmark Granularity and Model Robustness for Image-Text Retrieval](https://mlsanity.com/paper/2407.15239/) (2024)
- [Black-box Attacks on Image Activity Prediction and its Natural Language Explanations](https://mlsanity.com/paper/2310.00503/) (2023)
- [Task-Agnostic Attacks Against Vision Foundation Models](https://mlsanity.com/paper/2503.03842/) (2025)
- [Light as Deception: GPT-driven Natural Relighting Against Vision-Language Pre-training Models](https://mlsanity.com/paper/2505.24227/) (2025)
- [Rectifying Adversarial Sample with Low Entropy Prior for Test-Time Defense](https://mlsanity.com/paper/2507.03427/) (2025)
- [Revisiting CroPA: A Reproducibility Study and Enhancements for Cross-Prompt Adversarial Transferability in Vision-Language Models](https://mlsanity.com/paper/2506.22982/) (2025)

## Abstract (arXiv)

In this work, we study the robustness of a CNN+RNN based image captioning system being subjected to adversarial noises. We propose to fool an image captioning system to generate some targeted partial captions for an image polluted by adversarial noises, even the targeted captions are totally irrelevant to the image content. A partial caption indicates that the words at some locations in this caption are observed, while words at other locations are not restricted.It is the first work to study exact adversarial attacks of targeted partial captions. Due to the sequential dependencies among words in a caption, we formulate the generation of adversarial noises for targeted partial captions as a structured output learning problem with latent variables. Both the generalized expectation maximization algorithm and structural SVMs with latent variables are then adopted to optimize the problem. The proposed methods generate very successful at-tacks to three popular CNN+RNN based image captioning models. Furthermore, the proposed attack methods are used to understand the inner mechanism of image captioning systems, providing the guidance to further improve automatic image captioning systems towards human captioning.