# Attention Hijacking in Trojan Transformers

arXiv: https://arxiv.org/abs/2208.04946 · HTML: https://mlsanity.com/paper/2208.04946/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 60 references that are in the map.
- First posted on arXiv 9 Aug 2022.

## Builds on (most cited first, 30 of 43)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain](https://mlsanity.com/paper/1708.06733/) (2017)
- [Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning](https://mlsanity.com/paper/1712.05526/) (2017)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://mlsanity.com/paper/1905.09418/) (2019)
- [Are Sixteen Heads Really Better than One?](https://mlsanity.com/paper/1905.10650/) (2019)
- [Universal adversarial perturbations](https://mlsanity.com/paper/1610.08401/) (2016)
- [Universal Adversarial Triggers for Attacking and Analyzing NLP](https://mlsanity.com/paper/1908.07125/) (2019)
- [Do Vision Transformers See Like Convolutional Neural Networks?](https://mlsanity.com/paper/2108.08810/) (2021)
- [Backdoor Learning: A Survey](https://mlsanity.com/paper/2007.08745/) (2020)
- [Hidden Trigger Backdoor Attacks](https://mlsanity.com/paper/1910.00033/) (2019)
- [Reflection Backdoor: A Natural Backdoor Attack on Deep Neural Networks](https://mlsanity.com/paper/2007.02343/) (2020)
- [Input-Aware Dynamic Backdoor Attack](https://mlsanity.com/paper/2010.08138/) (2020)
- [How Do Vision Transformers Work?](https://mlsanity.com/paper/2202.06709/) (2022)
- [ONION: A Simple and Effective Defense Against Textual Backdoor Attacks](https://mlsanity.com/paper/2011.10369/) (2020)
- [Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth](https://mlsanity.com/paper/2010.15327/) (2020)
- [A backdoor attack against LSTM-based text classification systems](https://mlsanity.com/paper/1905.12457/) (2019)
- [Clean-Label Backdoor Attacks on Video Recognition Models](https://mlsanity.com/paper/2003.03030/) (2020)
- [Universal Litmus Patterns: Revealing Backdoor Attacks in CNNs](https://mlsanity.com/paper/1906.10842/) (2019)
- [Backdoor Attacks Against Deep Learning Systems in the Physical World](https://mlsanity.com/paper/2006.14580/) (2020)
- [Self-Attention Attribution: Interpreting Information Interactions Inside Transformer](https://mlsanity.com/paper/2004.11207/) (2020)
- [Be Careful about Poisoned Word Embeddings: Exploring the Vulnerability of the Embedding Layers in NLP Models](https://mlsanity.com/paper/2103.15543/) (2021)
- [RAP: Robustness-Aware Perturbations for Defending against Backdoor Attacks on NLP Models](https://mlsanity.com/paper/2110.07831/) (2021)
- [Backdoor Scanning for Deep Neural Networks through K-Arm Optimization](https://mlsanity.com/paper/2102.05123/) (2021)

## Built on by (most cited first, 6 of 6)

- [TrojVLM: Backdoor Attack Against Vision Language Models](https://mlsanity.com/paper/2409.19232/) (2024)
- [Comprehensive evaluation of Mal-API-2019 dataset by machine learning in malware detection](https://mlsanity.com/paper/2403.02232/) (2024)
- [BadCLM: Backdoor Attack in Clinical Language Models for Electronic Health Records](https://mlsanity.com/paper/2407.05213/) (2024)
- [Towards SAR Automatic Target Recognition MultiCategory SAR Image Classification Based on Light Weight Vision Transformer](https://mlsanity.com/paper/2407.06128/) (2024)
- [Backdoors in DRL: Four Environments Focusing on In-distribution Triggers](https://mlsanity.com/paper/2505.17248/) (2025)
- [Backdoor Learning in Language Models and Vision-Language Models](https://mlsanity.com/paper/2608.18095/) (2026)

## Abstract (arXiv)

Trojan attacks pose a severe threat to AI systems. Recent works on Transformer models received explosive popularity and the self-attentions are now indisputable. This raises a central question: Can we reveal the Trojans through attention mechanisms in BERTs and ViTs? In this paper, we investigate the attention hijacking pattern in Trojan AIs, ie, the trigger token ``kidnaps'' the attention weights when a specific trigger is present. We observe the consistent attention hijacking pattern in Trojan Transformers from both Natural Language Processing (NLP) and Computer Vision (CV) domains. This intriguing property helps us to understand the Trojan mechanism in BERTs and ViTs. We also propose an Attention-Hijacking Trojan Detector (AHTD) to discriminate the Trojan AIs from the clean ones.