# Towards Practical Certifiable Patch Defense with Vision Transformer

arXiv: https://arxiv.org/abs/2203.08519 · HTML: https://mlsanity.com/paper/2203.08519/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 48 references that are in the map.
- First posted on arXiv 16 Mar 2022.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [On Adaptive Attacks to Adversarial Example Defenses](https://mlsanity.com/paper/2002.08347/) (2020)
- [Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet](https://mlsanity.com/paper/1904.00760/) (2019)
- [Making an Invisibility Cloak: Real World Adversarial Attacks on Object Detectors](https://mlsanity.com/paper/1910.14667/) (2019)
- [LaVAN: Localized and Visible Adversarial Noise](https://mlsanity.com/paper/1801.02608/) (2018)
- [Certified Defenses for Adversarial Patches](https://mlsanity.com/paper/2003.06693/) (2020)
- [(De)Randomized Smoothing for Certifiable Defense against Patch Attacks](https://mlsanity.com/paper/2002.10733/) (2020)
- [Universal Physical Camouflage Attacks on Object Detectors](https://mlsanity.com/paper/1909.04326/) (2019)
- [Local Gradients Smoothing: Defense against localized adversarial attacks](https://mlsanity.com/paper/1807.01216/) (2018)
- [Efficient Image Super-Resolution Using Pixel Attention](https://mlsanity.com/paper/2010.01073/) (2020)
- [ClassSR: A General Framework to Accelerate Super-Resolution Networks by Data Characteristic](https://mlsanity.com/paper/2103.04039/) (2021)
- [CMUA-Watermark: A Cross-Model Universal Adversarial Watermark for Combating Deepfakes](https://mlsanity.com/paper/2105.10872/) (2021)
- [DPT: Deformable Patch-based Transformer for Visual Recognition](https://mlsanity.com/paper/2107.14467/) (2021)
- [Reflash Dropout in Image Super-Resolution](https://mlsanity.com/paper/2112.12089/) (2021)
- [Disentangled High Quality Salient Object Detection](https://mlsanity.com/paper/2108.03551/) (2021)
- [Efficient Certified Defenses Against Patch Attacks on Image Classifiers](https://mlsanity.com/paper/2102.04154/) (2021)
- [RPATTACK: Refined Patch Attack on General Object Detectors](https://mlsanity.com/paper/2103.12469/) (2021)
- [Efficient universal shuffle attack for visual object tracking](https://mlsanity.com/paper/2203.06898/) (2022)
- [PatchGuard: A Provably Robust Defense against Adversarial Patches via Small Receptive Fields and Masking](https://mlsanity.com/paper/2005.10884/) (2020)
- [Highly Efficient Natural Image Matting](https://mlsanity.com/paper/2110.12748/) (2021)

## Built on by (most cited first, 30 of 44)

- [Federated Learning with Label Distribution Skew via Logits Calibration](https://mlsanity.com/paper/2209.00189/) (2022)
- [Content-based Unrestricted Adversarial Attack](https://mlsanity.com/paper/2305.10665/) (2023)
- [How Deep Learning Sees the World: A Survey on Adversarial Attacks & Defenses](https://mlsanity.com/paper/2305.10862/) (2023)
- [T-SEA: Transfer-based Self-Ensemble Attack on Object Detection](https://mlsanity.com/paper/2211.09773/) (2022)
- [Generalized Video Anomaly Event Detection: Systematic Taxonomy and Comparison of Deep Models](https://mlsanity.com/paper/2302.05087/) (2023)
- [AIDE: A Vision-Driven Multi-View, Multi-Modal, Multi-Tasking Dataset for Assistive Driving Perception](https://mlsanity.com/paper/2307.13933/) (2023)
- [Context De-confounded Emotion Recognition](https://mlsanity.com/paper/2303.11921/) (2023)
- [Generative Domain Adaptation for Face Anti-Spoofing](https://mlsanity.com/paper/2207.10015/) (2022)
- [Boosting the Transferability of Adversarial Attacks with Global Momentum Initialization](https://mlsanity.com/paper/2211.11236/) (2022)
- [Contrastive Pseudo Learning for Open-World DeepFake Attribution](https://mlsanity.com/paper/2309.11132/) (2023)
- [DFRD: Data-Free Robustness Distillation for Heterogeneous Federated Learning](https://mlsanity.com/paper/2309.13546/) (2023)
- [Efficient Decision-based Black-box Patch Attacks on Video Recognition](https://mlsanity.com/paper/2303.11917/) (2023)
- [Delving into the Adversarial Robustness of Federated Learning](https://mlsanity.com/paper/2302.09479/) (2023)
- [Query-Efficient Decision-based Black-Box Patch Attack](https://mlsanity.com/paper/2307.00477/) (2023)
- [Certified Defences Against Adversarial Patch Attacks on Semantic Segmentation](https://mlsanity.com/paper/2209.05980/) (2022)
- [Improving Generalization in Visual Reinforcement Learning via Conflict-aware Gradient Agreement Augmentation](https://mlsanity.com/paper/2308.01194/) (2023)
- [Visual Adversarial Attacks and Defenses in the Physical World: A Survey](https://mlsanity.com/paper/2211.01671/) (2022)
- [PatchCURE: Improving Certifiable Robustness, Model Utility, and Computation Efficiency of Adversarial Patch Defenses](https://mlsanity.com/paper/2310.13076/) (2023)
- [De-confounded Data-free Knowledge Distillation for Handling Distribution Shifts](https://mlsanity.com/paper/2403.19539/) (2024)
- [Adversarial Examples in the Physical World: A Survey](https://mlsanity.com/paper/2311.01473/) (2023)
- [Adversarial Contrastive Distillation with Adaptive Denoising](https://mlsanity.com/paper/2302.08764/) (2023)
- [Explicit and Implicit Knowledge Distillation via Unlabeled Data](https://mlsanity.com/paper/2302.08771/) (2023)
- [Sampling to Distill: Knowledge Transfer from Open-World Data](https://mlsanity.com/paper/2307.16601/) (2023)
- [Out of Thin Air: Exploring Data-Free Adversarial Robustness Distillation](https://mlsanity.com/paper/2303.11611/) (2023)
- [A Survey on Physical Adversarial Attacks against Face Recognition Systems](https://mlsanity.com/paper/2410.16317/) (2024)
- [Real-Time Robust Video Object Detection System Against Physical-World Adversarial Attacks](https://mlsanity.com/paper/2208.09195/) (2022)
- [CrossCert: A Cross-Checking Detection Approach to Patch Robustness Certification for Deep Learning Models](https://mlsanity.com/paper/2405.07668/) (2024)
- [Self-Cooperation Knowledge Distillation for Novel Class Discovery](https://mlsanity.com/paper/2407.01930/) (2024)
- [A Majority Invariant Approach to Patch Robustness Certification for Deep Learning Models](https://mlsanity.com/paper/2308.00452/) (2023)
- [Towards Robust Semantic Segmentation against Patch-based Attack via Attention Refinement](https://mlsanity.com/paper/2401.01750/) (2024)

## Abstract (arXiv)

Patch attacks, one of the most threatening forms of physical attack in adversarial examples, can lead networks to induce misclassification by modifying pixels arbitrarily in a continuous region. Certifiable patch defense can guarantee robustness that the classifier is not affected by patch attacks. Existing certifiable patch defenses sacrifice the clean accuracy of classifiers and only obtain a low certified accuracy on toy datasets. Furthermore, the clean and certified accuracy of these methods is still significantly lower than the accuracy of normal classification networks, which limits their application in practice. To move towards a practical certifiable patch defense, we introduce Vision Transformer (ViT) into the framework of Derandomized Smoothing (DS). Specifically, we propose a progressive smoothed image modeling task to train Vision Transformer, which can capture the more discriminable local context of an image while preserving the global semantic information. For efficient inference and deployment in the real world, we innovatively reconstruct the global self-attention structure of the original ViT into isolated band unit self-attention. On ImageNet, under 2% area patch attacks our method achieves 41.70% certified accuracy, a nearly 1-fold increase over the previous best method (26.00%). Simultaneously, our method achieves 78.58% clean accuracy, which is quite close to the normal ResNet-101 accuracy. Extensive experiments show that our method obtains state-of-the-art clean and certified accuracy with inferring efficiently on CIFAR-10 and ImageNet.