# TrojViT: Trojan Insertion in Vision Transformers

arXiv: https://arxiv.org/abs/2208.13049 · HTML: https://mlsanity.com/paper/2208.13049/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security.
- Builds on 23 of its 35 references that are in the map.
- First posted on arXiv 27 Aug 2022.

## Placed in

- LLM safety and security / [Backdoor and data poisoning attacks](https://mlsanity.com/safety/backdoor-and-data-poisoning-attacks/)

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain](https://mlsanity.com/paper/1708.06733/) (2017)
- [Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning](https://mlsanity.com/paper/1712.05526/) (2017)
- [Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks](https://mlsanity.com/paper/1805.12185/) (2018)
- [Detecting Backdoor Attacks on Deep Neural Networks by Activation Clustering](https://mlsanity.com/paper/1811.03728/) (2018)
- [Input-Aware Dynamic Backdoor Attack](https://mlsanity.com/paper/2010.08138/) (2020)
- [Label-Consistent Backdoor Attacks](https://mlsanity.com/paper/1912.02771/) (2019)
- [A new Backdoor Attack in CNNs by training set corruption without label poisoning](https://mlsanity.com/paper/1902.11237/) (2019)
- [Language model compression with weighted low-rank factorization](https://mlsanity.com/paper/2207.00112/) (2022)
- [Backdoor Attacks Against Deep Learning Systems in the Physical World](https://mlsanity.com/paper/2006.14580/) (2020)
- [TBT: Targeted Neural Network Attack with Bit Trojan](https://mlsanity.com/paper/1909.05193/) (2019)
- [BppAttack: Stealthy and Efficient Trojan Attacks against Deep Neural Networks via Image Quantization and Contrastive Adversarial Learning](https://mlsanity.com/paper/2205.13383/) (2022)
- [Hermes Attack: Steal DNN Models with Lossless Inference Accuracy](https://mlsanity.com/paper/2006.12784/) (2020)
- [Imperceptible Backdoor Attack: From Input Space to Feature Representation](https://mlsanity.com/paper/2205.03190/) (2022)
- [Robust Backdoor Attacks against Deep Neural Networks in Real Physical World](https://mlsanity.com/paper/2104.07395/) (2021)
- [Defending Backdoor Attacks on Vision Transformer via Patch Processing](https://mlsanity.com/paper/2206.12381/) (2022)
- [RowHammer: A Retrospective](https://mlsanity.com/paper/1904.09724/) (2019)
- [Backdoor Attacks on Vision Transformers](https://mlsanity.com/paper/2206.08477/) (2022)
- [ADA-Tucker: Compressing Deep Neural Networks via Adaptive Dimension Adjustment Tucker Decomposition](https://mlsanity.com/paper/1906.07671/) (2019)
- [DBIA: Data-free Backdoor Injection Attack against Transformer Networks](https://mlsanity.com/paper/2111.11870/) (2021)

## Built on by (most cited first, 30 of 31)

- [BAGM: A Backdoor Attack for Manipulating Text-to-Image Generative Models](https://mlsanity.com/paper/2307.16489/) (2023)
- [TrojLLM: A Black-box Trojan Prompt Attack on Large Language Models](https://mlsanity.com/paper/2306.06815/) (2023)
- [TrojText: Test-time Invisible Textual Trojan Insertion](https://mlsanity.com/paper/2303.02242/) (2023)
- [DisDet: Exploring Detectability of Backdoor Attack on Diffusion Models](https://mlsanity.com/paper/2402.02739/) (2024)
- [Quantifying Bias in Text-to-Image Generative Models](https://mlsanity.com/paper/2312.13053/) (2023)
- [Mitigating the Backdoor Effect for Multi-Task Model Merging via Safety-Aware Subspace](https://mlsanity.com/paper/2410.13910/) (2024)
- [SSL-Cleanse: Trojan Detection and Mitigation in Self-Supervised Learning](https://mlsanity.com/paper/2303.09079/) (2023)
- [CipherPrune: Efficient and Scalable Private Transformer Inference](https://mlsanity.com/paper/2502.16782/) (2025)
- [Erased but Not Forgotten: How Backdoors Compromise Concept Erasure](https://mlsanity.com/paper/2504.21072/) (2025)
- [Not All Prompts Are Secure: A Switchable Backdoor Attack Against Pre-trained Vision Transformers](https://mlsanity.com/paper/2405.10612/) (2024)
- [PrisonBreak: Jailbreaking Large Language Models with at Most Twenty-Five Targeted Bit-flips](https://mlsanity.com/paper/2412.07192/) (2024)
- [Backdoor in Seconds: Unlocking Vulnerabilities in Large Pre-trained Models via Model Editing](https://mlsanity.com/paper/2410.18267/) (2024)
- [TrojFair: Trojan Fairness Attacks](https://mlsanity.com/paper/2312.10508/) (2023)
- [Context is the Key: Backdoor Attacks for In-Context Learning with Vision Transformers](https://mlsanity.com/paper/2409.04142/) (2024)
- [Backdoor Attack Against Vision Transformers via Attention Gradient-Based Image Erosion](https://mlsanity.com/paper/2410.22678/) (2024)
- [Backdoors in DRL: Four Environments Focusing on In-distribution Triggers](https://mlsanity.com/paper/2505.17248/) (2025)
- [DF-LoGiT: Data-Free Logic-Gated Backdoor Attacks in Vision Transformers](https://mlsanity.com/paper/2602.03040/) (2026)
- [Conjunctive Prompt Attacks in Multi-Agent LLM Systems](https://mlsanity.com/paper/2604.16543/) (2026)
- [Hiding Backdoors within Event Sequence Data via Poisoning Attacks](https://mlsanity.com/paper/2308.10201/) (2023)
- [Using Interleaved Ensemble Unlearning to Keep Backdoors at Bay for Finetuning Vision Transformers](https://mlsanity.com/paper/2410.01128/) (2024)
- [Task-based Loss Functions in Computer Vision: A Comprehensive Review](https://mlsanity.com/paper/2504.04242/) (2025)
- [Targeted Bit-Flip Attacks on LLM-Based Agents](https://mlsanity.com/paper/2603.10042/) (2026)
- [Backdoor Directions in Vision Transformers](https://mlsanity.com/paper/2603.10806/) (2026)
- [Recover Triggered States: Protect Model Against Backdoor Attack in Reinforcement Learning](https://mlsanity.com/paper/2304.00252/) (2023)
- [Manipulating and Mitigating Generative Model Biases without Retraining](https://mlsanity.com/paper/2404.02530/) (2024)
- [An Effective and Resilient Backdoor Attack Framework against Deep Neural Networks and Vision Transformers](https://mlsanity.com/paper/2412.06149/) (2024)
- [BadViM: Backdoor Attack against Vision Mamba](https://mlsanity.com/paper/2507.00577/) (2025)
- [PASTA: A Patch-Agnostic Twofold-Stealthy Backdoor Attack on Vision Transformers](https://mlsanity.com/paper/2604.20047/) (2026)
- [Exposing Functional Fusion: A New Class of Strategic Backdoor in Dynamic Prompt Architectures](https://mlsanity.com/paper/2605.19478/) (2026)
- [TEE-X: TEE-aware Acceleration Framework for Large Vision Models at the Edge](https://mlsanity.com/paper/2608.22716/) (2026)

## Abstract (arXiv)

Vision Transformers (ViTs) have demonstrated the state-of-the-art performance in various vision-related tasks. The success of ViTs motivates adversaries to perform backdoor attacks on ViTs. Although the vulnerability of traditional CNNs to backdoor attacks is well-known, backdoor attacks on ViTs are seldom-studied. Compared to CNNs capturing pixel-wise local features by convolutions, ViTs extract global context information through patches and attentions. Na"ively transplanting CNN-specific backdoor attacks to ViTs yields only a low clean data accuracy and a low attack success rate. In this paper, we propose a stealth and practical ViT-specific backdoor attack TrojViT. Rather than an area-wise trigger used by CNN-specific backdoor attacks, TrojViT generates a patch-wise trigger designed to build a Trojan composed of some vulnerable bits on the parameters of a ViT stored in DRAM memory through patch salience ranking and attention-target loss. TrojViT further uses minimum-tuned parameter update to reduce the bit number of the Trojan. Once the attacker inserts the Trojan into the ViT model by flipping the vulnerable bits, the ViT model still produces normal inference accuracy with benign inputs. But when the attacker embeds a trigger into an input, the ViT model is forced to classify the input to a predefined target class. We show that flipping only few vulnerable bits identified by TrojViT on a ViT model using the well-known RowHammer can transform the model into a backdoored one. We perform extensive experiments of multiple datasets on various ViT models. TrojViT can classify 99.64% of test images to a target class by flipping 345 bits on a ViT for ImageNet.Our codes are available at https://github.com/mxzheng/TrojViT