mlsanityData to 7 Oct 2026

Decision-based Black-box Attack Against Vision Transformers via Patch-wise Adversarial Removal

arXiv 2112.03492 ↗First posted 7 Dec 2021cs.CV cs.CRcited by 16

Not on the map. Why?

Abstract (arXiv)

Vision transformers (ViTs) have demonstrated impressive performance and stronger adversarial robustness compared to Convolutional Neural Networks (CNNs). On the one hand, ViTs' focus on global interaction between individual patches reduces the local noise sensitivity of images. On the other hand, the neglect of noise sensitivity differences between image regions by existing decision-based attacks further compromises the efficiency of noise compression, especially for ViTs. Therefore, validating the black-box adversarial robustness of ViTs when the target model can only be queried still remains a challenging problem. In this paper, we theoretically analyze the limitations of existing decision-based attacks from the perspective of noise sensitivity difference between regions of the image, and propose a new decision-based black-box attack against ViTs, termed Patch-wise Adversarial Removal (PAR). PAR divides images into patches through a coarse-to-fine search process and compresses the noise on each patch separately. PAR records the noise magnitude and noise sensitivity of each patch and selects the patch with the highest query value for noise compression. In addition, PAR can be used as a noise initialization method for other decision-based attacks to improve the noise compression efficiency on both ViTs and CNNs without introducing additional calculations. Extensive experiments on three datasets demonstrate that PAR achieves a much lower noise magnitude with the same number of queries.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 36 of its 53 references that are in the map. First posted on arXiv 7 Dec 2021.

Builds on

36 of 53 references.

References · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  7. 2016Densely Connected Convolutional Networkscited by 6,691
  8. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  9. 2013Intriguing properties of neural networkscited by 5,114
  10. 2017Squeeze-and-Excitation Networkscited by 4,763
  11. 2016Adversarial examples in the physical worldcited by 1,762
  12. 2016Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learningcited by 1,509
24 more
  1. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
  2. 2021Video Swin Transformercited by 1,090
  3. 2016Practical Black-Box Attacks against Machine Learningcited by 893
  4. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  5. 2014Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Imagescited by 774
  6. 2017One pixel attack for fooling deep neural networkscited by 579
  7. 2017Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Modelscited by 459
  8. 2021Intriguing Properties of Vision Transformerscited by 446
  9. 2019Evading Defenses to Transferable Adversarial Examples by Translation-Invariant Attackscited by 440
  10. 2019HopSkipJumpAttack: A Query-Efficient Decision-Based Attackcited by 328
  11. 2021Understanding Robustness of Transformers for Image Classificationcited by 250
  12. 2021Vision Transformers are Robust Learnerscited by 208
  13. 2019Efficient Decision-based Black-box Adversarial Attacks on Face Recognitioncited by 173
  14. 2021Towards Robust Vision Transformercited by 137
  15. 2019Sign-OPT: A Query-Efficient Hard-label Adversarial Attackcited by 118
  16. 2021Towards Transferable Adversarial Attacks on Vision Transformerscited by 80
  17. 2021On Improving Adversarial Transferability of Vision Transformerscited by 70
  18. 2020SurFree: a fast surrogate-free black-box attackcited by 59
  19. 2018Guessing Smart: Biased Sampling for Efficient Black-Box Adversarial Attackscited by 46
  20. 2021Reveal of Vision Transformers Robustness against Adversarial Attackscited by 43
  21. 2018Understanding and Enhancing the Transferability of Adversarial Examplescited by 41
  22. 2021Attack to Fool and Explain Deep Networkscited by 17
  23. 2018Adversarial Vision Challengecited by 9
  24. 2021LSDAT: Low-Rank and Sparse Decomposition for Decision-based Adversarial Attackcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 5; 2024: 5; 2025: 2; 2026: 3.

16 papers.

Sort:

By area:

  1. 2022Towards Efficient Adversarial Training on Vision Transformerscited by 29
  2. 2024B-AVIBench: Towards Evaluating the Robustness of Large Vision-Language Model on Black-box Adversarial Visual-Instructionscited by 26
  3. 2023Robustifying Token Attention for Vision Transformerscited by 19
  4. 2023Adventures of Trustworthy Vision-Language Models: A Surveycited by 10
  5. 2025Towards Model Resistant to Transferable Adversarial Examples via Trigger Activationcited by 10
  6. 2023It Is All About Data: A Survey on the Effects of Data on Adversarial Robustnesscited by 9
  7. 2024ADBA:Approximation Decision Boundary Approach for Black-Box Adversarial Attackscited by 7
  8. 2024Attacking Transformers with Feature Diversity Adversarial Perturbationcited by 3
  9. 2023Random Position Adversarial Patch for Vision Transformerscited by 2
  10. 2024Query-Efficient Hard-Label Black-Box Attack against Vision Transformerscited by 2
  11. 2024ViTGuard: Attention-aware Detection against Adversarial Examples for Vision Transformercited by 2
  12. 2025CertMask: Certifiable Defense Against Adversarial Patches via Theoretically Optimal Mask Coveragecited by 2
4 more
  1. 2023Exploring Non-additive Randomness on ViT against Query-Based Black-Box Attackscited by 1
  2. 2026Contract And Conquer: How to Provably Compute Adversarial Examples for a Black-Box Model?cited by 0
  3. 2026PHANTOM: A Large-Scale Dataset of Multimodal Adversarial Attacks for Vision-Language Modelscited by 0
  4. 2026TEE-X: TEE-aware Acceleration Framework for Large Vision Models at the Edgecited by 0