# Defending Backdoor Attacks on Vision Transformer via Patch Processing

arXiv: https://arxiv.org/abs/2206.12381 · HTML: https://mlsanity.com/paper/2206.12381/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security.
- Builds on 45 of its 66 references that are in the map.
- First posted on arXiv 24 Jun 2022.

## Placed in

- LLM safety and security / [Backdoor and data poisoning attacks](https://mlsanity.com/safety/backdoor-and-data-poisoning-attacks/)

## Builds on (most cited first, 30 of 45)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning](https://mlsanity.com/paper/1712.05526/) (2017)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Big Transfer (BiT): General Visual Representation Learning](https://mlsanity.com/paper/1912.11370/) (2019)
- [Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks](https://mlsanity.com/paper/1805.12185/) (2018)
- [Spectral Signatures in Backdoor Attacks](https://mlsanity.com/paper/1811.00636/) (2018)
- [Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey](https://mlsanity.com/paper/1801.00553/) (2018)
- [STRIP: A Defence Against Trojan Attacks on Deep Neural Networks](https://mlsanity.com/paper/1902.06531/) (2019)
- [Detecting Backdoor Attacks on Deep Neural Networks by Activation Clustering](https://mlsanity.com/paper/1811.03728/) (2018)
- [Intriguing Properties of Vision Transformers](https://mlsanity.com/paper/2105.10497/) (2021)
- [A Survey of Deep Learning Techniques for Autonomous Driving](https://mlsanity.com/paper/1910.07738/) (2019)
- [Hidden Trigger Backdoor Attacks](https://mlsanity.com/paper/1910.00033/) (2019)
- [Reflection Backdoor: A Natural Backdoor Attack on Deep Neural Networks](https://mlsanity.com/paper/2007.02343/) (2020)
- [Understanding Robustness of Transformers for Image Classification](https://mlsanity.com/paper/2103.14586/) (2021)
- [A new Backdoor Attack in CNNs by training set corruption without label poisoning](https://mlsanity.com/paper/1902.11237/) (2019)
- [Blind Backdoors in Deep Learning Models](https://mlsanity.com/paper/2005.03823/) (2020)
- [On the Robustness of Vision Transformers to Adversarial Examples](https://mlsanity.com/paper/2104.02610/) (2021)
- [Neural Trojans](https://mlsanity.com/paper/1710.00942/) (2017)
- [Towards Robust Vision Transformer](https://mlsanity.com/paper/2105.07926/) (2021)
- [Defending Neural Backdoors via Generative Distribution Modeling](https://mlsanity.com/paper/1910.04749/) (2019)
- [Rethinking the Trigger of Backdoor Attack](https://mlsanity.com/paper/2004.04692/) (2020)
- [DeepSweep: An Evaluation Framework for Mitigating DNN Backdoor Attacks using Data Augmentation](https://mlsanity.com/paper/2012.07006/) (2020)

## Built on by (most cited first, 20 of 20)

- [Marksman Backdoor: Backdoor Attacks with Arbitrary Target Class](https://mlsanity.com/paper/2210.09194/) (2022)
- [TrojViT: Trojan Insertion in Vision Transformers](https://mlsanity.com/paper/2208.13049/) (2022)
- [DisDet: Exploring Detectability of Backdoor Attack on Diffusion Models](https://mlsanity.com/paper/2402.02739/) (2024)
- [TextGuard: Provable Defense against Backdoor Attacks on Text Classification](https://mlsanity.com/paper/2311.11225/) (2023)
- [Boosting Backdoor Attack with A Learnable Poisoning Sample Selection Strategy](https://mlsanity.com/paper/2307.07328/) (2023)
- [Not All Prompts Are Secure: A Switchable Backdoor Attack Against Pre-trained Vision Transformers](https://mlsanity.com/paper/2405.10612/) (2024)
- [Perturb and Recover: Fine-tuning for Effective Backdoor Removal from CLIP](https://mlsanity.com/paper/2412.00727/) (2024)
- [Backdoor Attack Against Vision Transformers via Attention Gradient-Based Image Erosion](https://mlsanity.com/paper/2410.22678/) (2024)
- [DF-LoGiT: Data-Free Logic-Gated Backdoor Attacks in Vision Transformers](https://mlsanity.com/paper/2602.03040/) (2026)
- [Degenerate Swin to Win: Plain Window-based Transformer without Sophisticated Operations](https://mlsanity.com/paper/2211.14255/) (2022)
- [Exploring Non-additive Randomness on ViT against Query-Based Black-Box Attacks](https://mlsanity.com/paper/2309.06438/) (2023)
- [TokenMark: A Modality-Agnostic Watermark for Pre-trained Transformers](https://mlsanity.com/paper/2403.05842/) (2024)
- [BadVim: Unveiling Backdoor Threats in Visual State Space Model](https://mlsanity.com/paper/2408.11679/) (2024)
- [Using Interleaved Ensemble Unlearning to Keep Backdoors at Bay for Finetuning Vision Transformers](https://mlsanity.com/paper/2410.01128/) (2024)
- [Backdoor Directions in Vision Transformers](https://mlsanity.com/paper/2603.10806/) (2026)
- [BackdoorIDS: Zero-shot Backdoor Detection for Pretrained Vision Encoder](https://mlsanity.com/paper/2603.11664/) (2026)
- [An Effective and Resilient Backdoor Attack Framework against Deep Neural Networks and Vision Transformers](https://mlsanity.com/paper/2412.06149/) (2024)
- [Backdoor Attacks on Open Vocabulary Object Detectors via Multi-Modal Prompt Tuning](https://mlsanity.com/paper/2511.12735/) (2025)
- [IU: Imperceptible Universal Backdoor Attack](https://mlsanity.com/paper/2603.00711/) (2026)
- [PASTA: A Patch-Agnostic Twofold-Stealthy Backdoor Attack on Vision Transformers](https://mlsanity.com/paper/2604.20047/) (2026)

## Abstract (arXiv)

Vision Transformers (ViTs) have a radically different architecture with significantly less inductive bias than Convolutional Neural Networks. Along with the improvement in performance, security and robustness of ViTs are also of great importance to study. In contrast to many recent works that exploit the robustness of ViTs against adversarial examples, this paper investigates a representative causative attack, i.e., backdoor. We first examine the vulnerability of ViTs against various backdoor attacks and find that ViTs are also quite vulnerable to existing attacks. However, we observe that the clean-data accuracy and backdoor attack success rate of ViTs respond distinctively to patch transformations before the positional encoding. Then, based on this finding, we propose an effective method for ViTs to defend both patch-based and blending-based trigger backdoor attacks via patch processing. The performances are evaluated on several benchmark datasets, including CIFAR10, GTSRB, and TinyImageNet, which show the proposed novel defense is very successful in mitigating backdoor attacks for ViTs. To the best of our knowledge, this paper presents the first defensive strategy that utilizes a unique characteristic of ViTs against backdoor attacks. The paper will appear in the Proceedings of the AAAI'23 Conference. This work was initially submitted in November 2021 to CVPR'22, then it was re-submitted to ECCV'22. The paper was made public in June 2022. The authors sincerely thank all the referees from the Program Committees of CVPR'22, ECCV'22, and AAAI'23.