# Trojan Signatures in DNN Weights

arXiv: https://arxiv.org/abs/2109.02836 · HTML: https://mlsanity.com/paper/2109.02836/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security.
- Builds on 7 of its 17 references that are in the map.
- First posted on arXiv 7 Sep 2021.

## Placed in

- LLM safety and security / [Backdoor and data poisoning attacks](https://mlsanity.com/safety/backdoor-and-data-poisoning-attacks/)

## Builds on (most cited first, 7 of 7)

- [BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain](https://mlsanity.com/paper/1708.06733/) (2017)
- [STRIP: A Defence Against Trojan Attacks on Deep Neural Networks](https://mlsanity.com/paper/1902.06531/) (2019)
- [Detecting Backdoor Attacks on Deep Neural Networks by Activation Clustering](https://mlsanity.com/paper/1811.03728/) (2018)
- [Practical Detection of Trojan Neural Networks: Data-Limited and Data-Free Cases](https://mlsanity.com/paper/2007.15802/) (2020)
- [The TrojAI Software Framework: An OpenSource tool for Embedding Trojans into Deep Learning Models](https://mlsanity.com/paper/2003.07233/) (2020)
- [Februus: Input Purification Defense Against Trojan Attacks on Deep Neural Network Systems](https://mlsanity.com/paper/1908.03369/) (2019)
- [CLEANN: Accelerated Trojan Shield for Embedded Neural Networks](https://mlsanity.com/paper/2009.02326/) (2020)

## Built on by (most cited first, 17 of 17)

- [Mitigating Backdoor Threats to Large Language Models: Advancement and Challenges](https://mlsanity.com/paper/2409.19993/) (2024)
- [Defenses in Adversarial Machine Learning: A Survey](https://mlsanity.com/paper/2312.08890/) (2023)
- [TIJO: Trigger Inversion with Joint Optimization for Defending Multimodal Backdoored Models](https://mlsanity.com/paper/2308.03906/) (2023)
- [Towards A Critical Evaluation of Robustness for Deep Learning Backdoor Countermeasures](https://mlsanity.com/paper/2204.06273/) (2022)
- [Rethinking Backdoor Detection Evaluation for Language Models](https://mlsanity.com/paper/2409.00399/) (2024)
- [Watch Out! Simple Horizontal Class Backdoor Can Trivially Evade Defense](https://mlsanity.com/paper/2310.00542/) (2023)
- [Gradient Shaping: Enhancing Backdoor Attack Against Reverse Engineering](https://mlsanity.com/paper/2301.12318/) (2023)
- [Measuring Impacts of Poisoning on Model Parameters and Embeddings for Large Language Models of Code](https://mlsanity.com/paper/2405.11466/) (2024)
- [NTD: Non-Transferability Enabled Backdoor Detection](https://mlsanity.com/paper/2111.11157/) (2021)
- [An Adaptive Black-box Backdoor Detection Method for Deep Neural Networks](https://mlsanity.com/paper/2204.04329/) (2022)
- [Measuring Impacts of Poisoning on Model Parameters and Neuron Activations: A Case Study of Poisoning CodeBERT](https://mlsanity.com/paper/2402.12936/) (2024)
- [On Trojan Signatures in Large Language Models of Code](https://mlsanity.com/paper/2402.16896/) (2024)
- [Solving Trojan Detection Competitions with Linear Weight Classification](https://mlsanity.com/paper/2411.03445/) (2024)
- [Trojan Cleansing with Neural Collapse](https://mlsanity.com/paper/2411.12914/) (2024)
- [MergeGuard: Efficient Thwarting of Trojan Attacks in Machine Learning Models](https://mlsanity.com/paper/2505.04015/) (2025)
- [Lightweight and Fast Backdoor Model Detection](https://mlsanity.com/paper/2605.18907/) (2026)
- [Fast and Lightweight Backdoor Detection via Head Random Probing](https://mlsanity.com/paper/2605.18908/) (2026)

## Abstract (arXiv)

Deep neural networks have been shown to be vulnerable to backdoor, or trojan, attacks where an adversary has embedded a trigger in the network at training time such that the model correctly classifies all standard inputs, but generates a targeted, incorrect classification on any input which contains the trigger. In this paper, we present the first ultra light-weight and highly effective trojan detection method that does not require access to the training/test data, does not involve any expensive computations, and makes no assumptions on the nature of the trojan trigger. Our approach focuses on analysis of the weights of the final, linear layer of the network. We empirically demonstrate several characteristics of these weights that occur frequently in trojaned networks, but not in benign networks. In particular, we show that the distribution of the weights associated with the trojan target class is clearly distinguishable from the weights associated with other classes. Using this, we demonstrate the effectiveness of our proposed detection method against state-of-the-art attacks across a variety of architectures, datasets, and trigger types.