Query-Efficient Hard-Label Black-Box Attack against Vision Transformers
Not on the map. Why?
Abstract (arXiv)
Recent studies have revealed that vision transformers (ViTs) face similar security risks from adversarial attacks as deep convolutional neural networks (CNNs). However, directly applying attack methodology on CNNs to ViTs has been demonstrated to be ineffective since the ViTs typically work on patch-wise encoding. This article explores the vulnerability of ViTs against adversarial attacks under a black-box scenario, and proposes a novel query-efficient hard-label adversarial attack method called AdvViT. Specifically, considering that ViTs are highly sensitive to patch modification, we propose to optimize the adversarial perturbation on the individual patches. To reduce the dimension of perturbation search space, we modify only a handful of low-frequency components of each patch. Moreover, we design a weight mask matrix for all patches to further optimize the perturbation on different regions of a whole image. We test six mainstream ViT backbones on the ImageNet-1k dataset. Experimental results show that compared with the state-of-the-art attacks on CNNs, our AdvViT achieves much lower L2-norm distortion under the same query budget, sufficiently validating the vulnerability of ViTs against adversarial attacks.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 46 of its 62 references that are in the map. First posted on arXiv 29 Jun 2024.
Builds on
46 of 62 references.
References · 46
- 2017Attention Is All You Needcited by 60,361
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2014Explaining and Harnessing Adversarial Examplescited by 7,364
- 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
- 2013Intriguing properties of neural networkscited by 5,155
- 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
- 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
- 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
34 more
- 2016Adversarial examples in the physical worldcited by 1,769
- 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,369
- 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,268
- 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
- 2021Video Swin Transformercited by 1,091
- 2021CvT: Introducing Convolutions to Vision Transformerscited by 947
- 2015The Limitations of Deep Learning in Adversarial Settingscited by 790
- 2017ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Modelscited by 666
- 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 527
- 2018Black-box Adversarial Attacks with Limited Queries and Informationcited by 501
- 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
- 2017Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Modelscited by 462
- 2021Early Convolutions Help Transformers See Bettercited by 396
- 2017Adversarial Examples: Attacks and Defenses for Deep Learningcited by 351
- 2019Simple Black-box Adversarial Attackscited by 251
- 2021Understanding Robustness of Transformers for Image Classificationcited by 249
- 2021Are Transformers More Robust Than CNNs?cited by 187
- 2021On the Robustness of Vision Transformers to Adversarial Examplescited by 167
- 2021Efficient Self-supervised Vision Transformers for Representation Learningcited by 149
- 2018AutoZOOM: Autoencoder-based Zeroth Order Optimization Method for Attacking Black-box Neural Networkscited by 137
- 2019Sign-OPT: A Query-Efficient Hard-label Adversarial Attackcited by 120
- 2021On the Adversarial Robustness of Vision Transformerscited by 106
- 2020QEBA: Query-Efficient Boundary-Based Blackbox Attackcited by 104
- 2020Toward Transformer-Based Object Detectioncited by 84
- 2018Low Frequency Adversarial Perturbationcited by 80
- 2021Towards Transferable Adversarial Attacks on Vision Transformerscited by 80
- 2020Sparse-RS: a versatile framework for query-efficient sparse black-box adversarial attackscited by 65
- 2017Adversarial Transformation Networks: Learning to Generate Adversarial Examplescited by 62
- 2020SurFree: a fast surrogate-free black-box attackcited by 60
- 2022Patch-Fool: Are Vision Transformers Always Robust Against Adversarial Perturbations?cited by 49
- 2020Learning Black-Box Attackers with Transferable Priors and Query Feedbackcited by 32
- 2021Decision-based Black-box Attack Against Vision Transformers via Patch-wise Adversarial Removalcited by 16
- 2022Object-Attentional Untargeted Adversarial Attackcited by 2
- 2023Improving Robustness for Vision Transformer with a Simple Dynamic Scanning Augmentationcited by 1
Built on by
Citing papers per year: 2026: 2.
2 papers.