mlsanityData to 14 Oct 2026

A Study of the Attention Abnormality in Trojaned BERTs

arXiv 2205.08305 ↗First posted 13 May 2022cs.CR cs.AI cs.LGcited by 48

Not on the map. Why?

Cited by 22 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Trojan attacks raise serious security concerns. In this paper, we investigate the underlying mechanism of Trojaned BERT models. We observe the attention focus drifting behavior of Trojaned models, i.e., when encountering an poisoned input, the trigger token hijacks the attention focus regardless of the context. We provide a thorough qualitative and quantitative analysis of this phenomenon, revealing insights into the Trojan mechanism. Based on the observation, we propose an attention-based Trojan detector to distinguish Trojaned models from clean ones. To the best of our knowledge, this is the first paper to analyze the Trojan mechanism and to develop a Trojan detector based on the transformer's attention.

Key facts. Cited by 48 papers in the map (data through 14 Oct 2026). Builds on 33 of its 49 references that are in the map. First posted on arXiv 13 May 2022.

Builds on

33 of 49 references.

References · 33

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  4. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  5. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  6. 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,106
  7. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,055
  8. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  9. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  10. 2016Universal adversarial perturbationscited by 784
  11. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 722
  12. 2019Hidden Trigger Backdoor Attackscited by 371
21 more
  1. 2020Reflection Backdoor: A Natural Backdoor Attack on Deep Neural Networkscited by 349
  2. 2020Input-Aware Dynamic Backdoor Attackcited by 313
  3. 2020ONION: A Simple and Effective Defense Against Textual Backdoor Attackscited by 263
  4. 2019A backdoor attack against LSTM-based text classification systemscited by 227
  5. 2020Clean-Label Backdoor Attacks on Video Recognition Modelscited by 169
  6. 2019Universal Litmus Patterns: Revealing Backdoor Attacks in CNNscited by 146
  7. 2020Backdoor Attacks Against Deep Learning Systems in the Physical Worldcited by 134
  8. 2020Self-Attention Attribution: Interpreting Information Interactions Inside Transformercited by 125
  9. 2021Be Careful about Poisoned Word Embeddings: Exploring the Vulnerability of the Embedding Layers in NLP Modelscited by 105
  10. 2021RAP: Robustness-Aware Perturbations for Defending against Backdoor Attacks on NLP Modelscited by 104
  11. 2021Backdoor Scanning for Deep Neural Networks through K-Arm Optimizationcited by 98
  12. 2020Practical Detection of Trojan Neural Networks: Data-Limited and Data-Free Casescited by 95
  13. 2021T-Miner: A Generative Approach to Defend Against Trojan Attacks on DNN-based Text Classificationcited by 60
  14. 2020Can Adversarial Weight Perturbations Inject Neural Backdoors?cited by 56
  15. 2020Universal Adversarial Attacks with Natural Triggers for Text Classificationcited by 51
  16. 2021Trigger Hunting with a Topological Prior for Trojan Detectioncited by 39
  17. 2021Topological Detection of Trojaned Neural Networkscited by 34
  18. 2020Poison Attacks against Text Datasets with Conditional Adversarially Regularized Autoencodercited by 32
  19. 2020BAAAN: Backdoor Attacks Against Autoencoder and GAN-Based Machine Learning Modelscited by 25
  20. 2021On the Distribution, Sparsity, and Inference-time Quantization of Attention Values in Transformerscited by 14
  21. 2020Live Trojan Attacks on Deep Neural Networkscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 4; 2023: 5; 2024: 28; 2025: 6; 2026: 5.

48 papers.

Sort:

By area:

  1. 2024Attention Tracker: Detecting Prompt Injection Attacks in LLMscited by 95
  2. 2024"Glue pizza and eat rocks" -- Exploiting Vulnerabilities in Retrieval-Augmented Generative Modelscited by 60
  3. 2023Backdoor Attacks and Countermeasures in Natural Language Processing Models: A Comprehensive Security Reviewcited by 39
  4. 2024TrojVLM: Backdoor Attack Against Vision Language Modelscited by 39
  5. 2024A Survey of Recent Backdoor Attacks and Defenses in Large Language Modelscited by 36
  6. 2024Time Series Modeling for Heart Rate Prediction: From ARIMA to Transformerscited by 35
  7. 2025textitAgents Under Siege: Breaking Pragmatic Multi-Agent LLM Systems with Optimized Prompt Attackscited by 34
  8. 2024Backdooring Vision-Language Models with Out-Of-Distribution Datacited by 33
  9. 2023Attention-Enhancing Backdoor Attacks Against BERT-based Modelscited by 32
  10. 2024APEER: Automatic Prompt Engineering Enhances Large Language Model Rerankingcited by 28
  11. 2023From Shortcuts to Triggers: Backdoor Defense with Denoised PoEcited by 27
  12. 2024Task-Agnostic Detector for Insertion-Based Backdoor Attackscited by 23
36 more
  1. 2024Deconstructing The Ethics of Large Language Models from Long-standing Issues to New-emerging Dilemmas: A Surveycited by 23
  2. 2024BadActs: A Universal Backdoor Defense in the Activation Spacecited by 21
  3. 2022Expose Backdoors on the Way: A Feature-Based Efficient Defense against Textual Backdoor Attackscited by 19
  4. 2023Setting the Trap: Capturing and Defeating Backdoors in Pretrained Language Models through Honeypotscited by 19
  5. 2024Image anomaly detection and prediction scheme based on SSA optimized ResNet50-BiGRU modelcited by 18
  6. 2022Backdoor Cleansing with Unlabeled Datacited by 17
  7. 2024Large Language Model Bias Mitigation from the Perspective of Knowledge Editingcited by 16
  8. 2024Mitigating Backdoor Threats to Large Language Models: Advancement and Challengescited by 15
  9. 2022A Survey on Backdoor Attack and Defense in Natural Language Processingcited by 14
  10. 2024XMeCap: Meme Caption Generation with Sub-Image Adaptabilitycited by 14
  11. 2026The Trigger in the Haystack: Extracting and Reconstructing LLM Backdoor Triggerscited by 14
  12. 2024Hadamard Adapter: An Extreme Parameter-Efficient Adapter Tuning Method for Pre-trained Language Modelscited by 13
  13. 2024An Open and Large-Scale Dataset for Multi-Modal Climate Change-aware Crop Yield Predictionscited by 11
  14. 2024Research on Driver Facial Fatigue Detection Based on Yolov8 Modelcited by 11
  15. 2024Optimizing Search Advertising Strategies: Integrating Reinforcement Learning with Generalized Second-Price Auctions for Enhanced Ad Ranking and Biddingcited by 9
  16. 2024Rethinking Backdoor Detection Evaluation for Language Modelscited by 9
  17. 2023UOR: Universal Backdoor Attacks on Pre-trained Language Modelscited by 8
  18. 2024Comprehensive evaluation of Mal-API-2019 dataset by machine learning in malware detectioncited by 8
  19. 2025Fed-HeLLo: Efficient Federated Foundation Model Fine-Tuning with Heterogeneous LoRA Allocationcited by 7
  20. 2026Purifying Generative LLMs from Backdoors without Prior Knowledge or Clean Referencecited by 7
  21. 2022Attention Hijacking in Trojan Transformerscited by 6
  22. 2024Can Contrastive Learning Refine Embeddingscited by 5
  23. 2024BadCLM: Backdoor Attack in Clinical Language Models for Electronic Health Recordscited by 5
  24. 2025Credit Risk Identification in Supply Chains Using Generative Adversarial Networkscited by 5
  25. 2025Pruning Strategies for Backdoor Defense in LLMscited by 5
  26. 2024Diffusion-Refined VQA Annotations for Semi-Supervised Gaze Followingcited by 4
  27. 2024Towards Robust Vision Transformer via Masked Adaptive Ensemblecited by 4
  28. 2024Debiasing Machine Unlearning with Counterfactual Examplescited by 3
  29. 2025Unmasking Backdoors: An Explainable Defense via Gradient-Attention Anomaly Scoring for Pre-trained Language Modelscited by 3
  30. 2026Merging Triggers, Breaking Backdoors: Defensive Poisoning for Instruction-Tuned Language Modelscited by 3
  31. 2025NLP Security and Ethics, in the Wildcited by 2
  32. 2024Defense Against Syntactic Textual Backdoor Attacks with Token Substitutioncited by 1
  33. 2024Real-Time Summarization of Twittercited by 1
  34. 2024Analyzing Multi-Head Attention on Trojan BERT Modelscited by 0
  35. 2026Backdoor Containment via Expert Quarantine and Shutdown in LLMscited by 0
  36. 2026Backdoor Purification for LoRA-Tuned LLMs via Null-Space Projectioncited by 0