mlsanityData to 7 Oct 2026

Exact Adversarial Attack to Image Captioning via Structured Output Learning with Latent Variables

arXiv 1905.04016 ↗First posted 10 May 2019cs.CV cs.AIcited by 26

Not on the map. Why?

Abstract (arXiv)

In this work, we study the robustness of a CNN+RNN based image captioning system being subjected to adversarial noises. We propose to fool an image captioning system to generate some targeted partial captions for an image polluted by adversarial noises, even the targeted captions are totally irrelevant to the image content. A partial caption indicates that the words at some locations in this caption are observed, while words at other locations are not restricted.It is the first work to study exact adversarial attacks of targeted partial captions. Due to the sequential dependencies among words in a caption, we formulate the generation of adversarial noises for targeted partial captions as a structured output learning problem with latent variables. Both the generalized expectation maximization algorithm and structural SVMs with latent variables are then adopted to optimize the problem. The proposed methods generate very successful at-tacks to three popular CNN+RNN based image captioning models. Furthermore, the proposed attack methods are used to understand the inner mechanism of image captioning systems, providing the guidance to further improve automatic image captioning systems towards human captioning.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 6; 2022: 2; 2023: 6; 2024: 8; 2025: 4.

26 papers.

Sort:

By area:

  1. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 281
  2. 2021Advances in adversarial attacks and defenses in computer vision: A surveycited by 127
  3. 2022Parallel Rectangle Flip Attack: A Query-based Black-box Attack against Object Detectioncited by 79
  4. 2021Targeted Attack against Deep Neural Networks via Flipping Limited Weight Bitscited by 49
  5. 2024An Image Is Worth 1000 Lies: Adversarial Transferability across Prompts on Vision-Language Modelscited by 40
  6. 2024Chain of Attack: On the Robustness of Vision-Language Models Against Transfer-Based Adversarial Attackscited by 37
  7. 2024Efficient Generation of Targeted and Transferable Adversarial Examples for Vision-Language Models Via Diffusion Modelscited by 36
  8. 2021Consensus Graph Representation Learning for Better Grounded Image Captioningcited by 28
  9. 2023Attacks in Adversarial Machine Learning: A Systematic Survey from the Life-cycle Perspectivecited by 19
  10. 2024MirrorCheck: Efficient Adversarial Defense for Vision-Language Modelscited by 19
  11. 2021MagDR: Mask-guided Detection and Reconstruction for Defending Deepfakescited by 17
  12. 2023Fast Adversarial Training with Smooth Convergencecited by 14
14 more
  1. 2024Instant Adversarial Purification with Adversarial Consistency Distillationcited by 14
  2. 2022Versatile Weight Attack via Flipping Limited Bitscited by 13
  3. 2021Controlled Caption Generation for Images Through Adversarial Attackscited by 9
  4. 2023Shilling Black-box Review-based Recommender Systems through Fake Review Generationcited by 8
  5. 2021Semantically Stealthy Adversarial Attacks against Segmentation Modelscited by 7
  6. 2024AttackGNN: Red-Teaming GNNs in Hardware Security Using Reinforcement Learningcited by 6
  7. 2023GSE: Group-wise Sparse and Explainable Adversarial Attackscited by 5
  8. 2024AICAttack: Adversarial Image Captioning Attack with Attention-Based Optimizationcited by 5
  9. 2024Benchmark Granularity and Model Robustness for Image-Text Retrievalcited by 5
  10. 2023Black-box Attacks on Image Activity Prediction and its Natural Language Explanationscited by 1
  11. 2025Task-Agnostic Attacks Against Vision Foundation Modelscited by 1
  12. 2025Light as Deception: GPT-driven Natural Relighting Against Vision-Language Pre-training Modelscited by 1
  13. 2025Rectifying Adversarial Sample with Low Entropy Prior for Test-Time Defensecited by 1
  14. 2025Revisiting CroPA: A Reproducibility Study and Enhancements for Cross-Prompt Adversarial Transferability in Vision-Language Modelscited by 0