# Universal Adversarial Attack on Attention and the Resulting Dataset DAmageNet

arXiv: https://arxiv.org/abs/2001.06325 · HTML: https://mlsanity.com/paper/2001.06325/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jan 2020.

## Built on by (most cited first, 30 of 45)

- [Adversarial Sticker: A Stealthy Attack Method in the Physical World](https://mlsanity.com/paper/2104.06728/) (2021)
- [Test-Time Backdoor Attacks on Multimodal Large Language Models](https://mlsanity.com/paper/2402.08577/) (2024)
- [Visual correspondence-based explanations improve AI robustness and human-AI team accuracy](https://mlsanity.com/paper/2208.00780/) (2022)
- [Transferable Adversarial Attack based on Integrated Gradients](https://mlsanity.com/paper/2205.13152/) (2022)
- [One-Pixel Shortcut: on the Learning Preference of Deep Neural Networks](https://mlsanity.com/paper/2205.12141/) (2022)
- [Delving into Data: Effectively Substitute Training for Black-box Attack](https://mlsanity.com/paper/2104.12378/) (2021)
- [Universal Adversarial Examples in Remote Sensing: Methodology and Benchmark](https://mlsanity.com/paper/2202.07054/) (2022)
- [Enhance the Visual Representation via Discrete Adversarial Training](https://mlsanity.com/paper/2209.07735/) (2022)
- [Adversarial Attack on Attackers: Post-Process to Mitigate Black-Box Score-Based Query Attacks](https://mlsanity.com/paper/2205.12134/) (2022)
- [Self-Ensemble Protection: Training Checkpoints Are Good Data Protectors](https://mlsanity.com/paper/2211.12005/) (2022)
- [Boosting 3D Adversarial Attacks with Attacking On Frequency](https://mlsanity.com/paper/2201.10937/) (2022)
- [Revisiting Knowledge Distillation: An Inheritance and Exploration Framework](https://mlsanity.com/paper/2107.00181/) (2021)
- [Transfer Attacks Revisited: A Large-Scale Empirical Study in Real Computer Vision Settings](https://mlsanity.com/paper/2204.04063/) (2022)
- [MMT-ARD: Multimodal Multi-Teacher Adversarial Distillation for Robust Vision-Language Models](https://mlsanity.com/paper/2511.17448/) (2025)
- [Robust Natural Language Understanding with Residual Attention Debiasing](https://mlsanity.com/paper/2305.17627/) (2023)
- [Transferable Physical Attack against Object Detection with Separable Attention](https://mlsanity.com/paper/2205.09592/) (2022)
- [Investigating Catastrophic Overfitting in Fast Adversarial Training: A Self-fitting Perspective](https://mlsanity.com/paper/2302.11963/) (2023)
- [Adversarial Attacks and Defenses in Machine Learning-Powered Networks: A Contemporary Survey](https://mlsanity.com/paper/2303.06302/) (2023)
- [A Comprehensive Study on the Robustness of Image Classification and Object Detection in Remote Sensing: Surveying and Benchmarking](https://mlsanity.com/paper/2306.12111/) (2023)
- [LRS: Enhancing Adversarial Transferability through Lipschitz Regularized Surrogate](https://mlsanity.com/paper/2312.13118/) (2023)
- [FG-UAP: Feature-Gathering Universal Adversarial Perturbation](https://mlsanity.com/paper/2209.13113/) (2022)
- [Be Careful with Rotation: A Uniform Backdoor Pattern for 3D Shape](https://mlsanity.com/paper/2211.16192/) (2022)
- [Hamming Similarity and Graph Laplacians for Class Partitioning and Adversarial Image Detection](https://mlsanity.com/paper/2305.01808/) (2023)
- [Query Attack by Multi-Identity Surrogates](https://mlsanity.com/paper/2105.15010/) (2021)
- [Adversarial Fine-tune with Dynamically Regulated Adversary](https://mlsanity.com/paper/2204.13232/) (2022)
- [Dual Graphs of Polyhedral Decompositions for the Detection of Adversarial Attacks](https://mlsanity.com/paper/2211.13305/) (2022)
- [Web Artifact Attacks Disrupt Vision Language Models](https://mlsanity.com/paper/2503.13652/) (2025)
- [Boosting Adversarial Transferability via Ensemble Non-Attention](https://mlsanity.com/paper/2511.08937/) (2025)
- [Dominant Patterns: Critical Features Hidden in Deep Neural Networks](https://mlsanity.com/paper/2105.15057/) (2021)
- [Towards Understanding and Harnessing the Effect of Image Transformation in Adversarial Detection](https://mlsanity.com/paper/2201.01080/) (2022)

## Abstract (arXiv)

Adversarial attacks on deep neural networks (DNNs) have been found for several years. However, the existing adversarial attacks have high success rates only when the information of the victim DNN is well-known or could be estimated by the structure similarity or massive queries. In this paper, we propose to Attack on Attention (AoA), a semantic property commonly shared by DNNs. AoA enjoys a significant increase in transferability when the traditional cross entropy loss is replaced with the attention loss. Since AoA alters the loss function only, it could be easily combined with other transferability-enhancement techniques and then achieve SOTA performance. We apply AoA to generate 50000 adversarial samples from ImageNet validation set to defeat many neural networks, and thus name the dataset as DAmageNet. 13 well-trained DNNs are tested on DAmageNet, and all of them have an error rate over 85%. Even with defenses or adversarial training, most models still maintain an error rate over 70% on DAmageNet. DAmageNet is the first universal adversarial dataset. It could be downloaded freely and serve as a benchmark for robustness testing and adversarial training.