# Mimic and Fool: A Task Agnostic Adversarial Attack

arXiv: https://arxiv.org/abs/1906.04606 · HTML: https://mlsanity.com/paper/1906.04606/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jun 2019.

## Built on by (most cited first, 11 of 11)

- [A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trends](https://mlsanity.com/paper/2407.07403/) (2024)
- [Gradient Harmonization in Unsupervised Domain Adaptation](https://mlsanity.com/paper/2408.00288/) (2024)
- [Ask, Attend, Attack: A Effective Decision-Based Black-Box Targeted Attack for Image-to-Text Models](https://mlsanity.com/paper/2408.08989/) (2024)
- [Object-oriented backdoor attack against image captioning](https://mlsanity.com/paper/2401.02600/) (2024)
- [Hybrid Reinforced Medical Report Generation with M-Linear Attention and Repetition Penalty](https://mlsanity.com/paper/2210.13729/) (2022)
- [Image Captioning via Dynamic Path Customization](https://mlsanity.com/paper/2406.00334/) (2024)
- [Black-box Attacks on Image Activity Prediction and its Natural Language Explanations](https://mlsanity.com/paper/2310.00503/) (2023)
- [Exact Feature Collisions in Neural Networks](https://mlsanity.com/paper/2205.15763/) (2022)
- [Robust Few-shot Learning Without Using any Adversarial Samples](https://mlsanity.com/paper/2211.01598/) (2022)
- [From text to multimodal: a survey of adversarial example generation in question answering systems](https://mlsanity.com/paper/2312.16156/) (2023)
- [CAPM: Fast and Robust Verification on Maxpool-based CNN via Dual Network](https://mlsanity.com/paper/2407.09550/) (2024)

## Abstract (arXiv)

At present, adversarial attacks are designed in a task-specific fashion. However, for downstream computer vision tasks such as image captioning, image segmentation etc., the current deep learning systems use an image classifier like VGG16, ResNet50, Inception-v3 etc. as a feature extractor. Keeping this in mind, we propose Mimic and Fool, a task agnostic adversarial attack. Given a feature extractor, the proposed attack finds an adversarial image which can mimic the image feature of the original image. This ensures that the two images give the same (or similar) output regardless of the task. We randomly select 1000 MSCOCO validation images for experimentation. We perform experiments on two image captioning models, Show and Tell, Show Attend and Tell and one VQA model, namely, end-to-end neural module network (N2NMN). The proposed attack achieves success rate of 74.0%, 81.0% and 87.1% for Show and Tell, Show Attend and Tell and N2NMN respectively. We also propose a slight modification to our attack to generate natural-looking adversarial images. In addition, we also show the applicability of the proposed attack for invertible architecture. Since Mimic and Fool only requires information about the feature extractor of the model, it can be considered as a gray-box attack.