# Topological Detection of Trojaned Neural Networks

arXiv: https://arxiv.org/abs/2106.06469 · HTML: https://mlsanity.com/paper/2106.06469/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 75 references that are in the map.
- First posted on arXiv 11 Jun 2021.

## Builds on (most cited first, 30 of 43)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain](https://mlsanity.com/paper/1708.06733/) (2017)
- [Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning](https://mlsanity.com/paper/1712.05526/) (2017)
- [STRIP: A Defence Against Trojan Attacks on Deep Neural Networks](https://mlsanity.com/paper/1902.06531/) (2019)
- [Detecting Backdoor Attacks on Deep Neural Networks by Activation Clustering](https://mlsanity.com/paper/1811.03728/) (2018)
- [Weight Poisoning Attacks on Pre-trained Models](https://mlsanity.com/paper/2004.06660/) (2020)
- [Label-Consistent Backdoor Attacks](https://mlsanity.com/paper/1912.02771/) (2019)
- [Persistence Images: A Stable Vector Representation of Persistent Homology](https://mlsanity.com/paper/1507.06217/) (2015)
- [Topology-Preserving Deep Image Segmentation](https://mlsanity.com/paper/1906.05404/) (2019)
- [Neural Trojans](https://mlsanity.com/paper/1710.00942/) (2017)
- [Universal Litmus Patterns: Revealing Backdoor Attacks in CNNs](https://mlsanity.com/paper/1906.10842/) (2019)
- [Backdoor Embedding in Convolutional Neural Network Models via Invisible Perturbation](https://mlsanity.com/paper/1808.10307/) (2018)
- [PersLay: A Neural Network Layer for Persistence Diagrams and New Graph Topological Signatures](https://mlsanity.com/paper/1904.09378/) (2019)
- [TABOR: A Highly Accurate Approach to Inspecting and Restoring Trojan Backdoors in AI Systems](https://mlsanity.com/paper/1908.01763/) (2019)
- [Ripser: efficient computation of Vietoris-Rips persistence barcodes](https://mlsanity.com/paper/1908.02518/) (2019)
- [Deep Learning with Topological Signatures](https://mlsanity.com/paper/1707.04041/) (2017)
- [Topology of deep neural networks](https://mlsanity.com/paper/2004.06093/) (2020)
- [Practical Detection of Trojan Neural Networks: Data-Limited and Data-Free Cases](https://mlsanity.com/paper/2007.15802/) (2020)
- [Technical Report: When Does Machine Learning FAIL? Generalized Transferability for Evasion and Poisoning Attacks](https://mlsanity.com/paper/1803.06975/) (2018)
- [Sliced Wasserstein Kernel for Persistence Diagrams](https://mlsanity.com/paper/1706.03358/) (2017)
- [A Topological Filter for Learning with Label Noise](https://mlsanity.com/paper/2012.04835/) (2020)
- [A Stable Multi-Scale Kernel for Topological Machine Learning](https://mlsanity.com/paper/1412.6821/) (2014)
- [Topology-Aware Segmentation Using Discrete Morse Theory](https://mlsanity.com/paper/2103.09992/) (2021)
- [Neural Persistence: A Complexity Measure for Deep Neural Networks Using Algebraic Topology](https://mlsanity.com/paper/1812.09764/) (2018)
- [Graph Filtration Learning](https://mlsanity.com/paper/1905.10996/) (2019)
- [A Topological Regularizer for Classifiers via Persistent Homology](https://mlsanity.com/paper/1806.10714/) (2018)

## Built on by (most cited first, 30 of 34)

- [Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks](https://mlsanity.com/paper/2207.13243/) (2022)
- [Position: Topological Deep Learning is the New Frontier for Relational Learning](https://mlsanity.com/paper/2402.08871/) (2024)
- [A Study of the Attention Abnormality in Trojaned BERTs](https://mlsanity.com/paper/2205.08305/) (2022)
- [UNICORN: A Unified Backdoor Trigger Inversion Framework](https://mlsanity.com/paper/2304.02786/) (2023)
- [Trigger Hunting with a Topological Prior for Trojan Detection](https://mlsanity.com/paper/2110.08335/) (2021)
- [NOTABLE: Transferable Backdoor Attacks Against Prompt-based NLP Models](https://mlsanity.com/paper/2305.17826/) (2023)
- [Attention-Enhancing Backdoor Attacks Against BERT-based Models](https://mlsanity.com/paper/2310.14480/) (2023)
- [Cycle Representation Learning for Inductive Relation Prediction](https://mlsanity.com/paper/2110.02510/) (2021)
- [Backdoor Attacks Against Dataset Distillation](https://mlsanity.com/paper/2301.01197/) (2023)
- [Defending Against Patch-based Backdoor Attacks on Self-Supervised Learning](https://mlsanity.com/paper/2304.01482/) (2023)
- [Neural Approximation of Graph Topological Features](https://mlsanity.com/paper/2201.12032/) (2022)
- [A Survey of Neural Trojan Attacks and Defenses in Deep Learning](https://mlsanity.com/paper/2202.07183/) (2022)
- [BEAGLE: Forensics of Deep Learning Backdoor Attack for Better Defense](https://mlsanity.com/paper/2301.06241/) (2023)
- [Defenses in Adversarial Machine Learning: A Survey](https://mlsanity.com/paper/2312.08890/) (2023)
- [Preference Poisoning Attacks on Reward Model Learning](https://mlsanity.com/paper/2402.01920/) (2024)
- [Backdoor Attack and Defense in Federated Generative Adversarial Network-based Medical Image Synthesis](https://mlsanity.com/paper/2210.10886/) (2022)
- [Check Your Other Door! Creating Backdoor Attacks in the Frequency Domain](https://mlsanity.com/paper/2109.05507/) (2021)
- [Don't FREAK Out: A Frequency-Inspired Approach to Detecting Backdoor Poisoned Samples in DNNs](https://mlsanity.com/paper/2303.13211/) (2023)
- [Look, Listen, and Attack: Backdoor Attacks Against Video Action Recognition](https://mlsanity.com/paper/2301.00986/) (2023)
- [Attention Hijacking in Trojan Transformers](https://mlsanity.com/paper/2208.04946/) (2022)
- [DeBackdoor: A Deductive Framework for Detecting Backdoor Attacks on Deep Models with Limited Data](https://mlsanity.com/paper/2503.21305/) (2025)
- [On the Convergence of Optimizing Persistent-Homology-Based Losses](https://mlsanity.com/paper/2206.02946/) (2022)
- [Mixup Barcodes: Quantifying Geometric-Topological Interactions between Point Clouds](https://mlsanity.com/paper/2402.15058/) (2024)
- [Model-Contrastive Learning for Backdoor Defense](https://mlsanity.com/paper/2205.04411/) (2022)
- [Mask and Restore: Blind Backdoor Defense at Test Time with Masked Autoencoder](https://mlsanity.com/paper/2303.15564/) (2023)
- [Be Persistent: Towards a Unified Solution for Mitigating Shortcuts in Deep Learning](https://mlsanity.com/paper/2402.11237/) (2024)
- [Trojan Cleansing with Neural Collapse](https://mlsanity.com/paper/2411.12914/) (2024)
- [Topology of Out-of-Distribution Examples in Deep Neural Networks](https://mlsanity.com/paper/2501.12522/) (2025)
- [CBPF: Filtering Poisoned Data Based on Composite Backdoor Attack](https://mlsanity.com/paper/2406.16125/) (2024)
- [Characterization of topological structures in different neural network architectures](https://mlsanity.com/paper/2407.06286/) (2024)

## Abstract (arXiv)

Deep neural networks are known to have security issues. One particular threat is the Trojan attack. It occurs when the attackers stealthily manipulate the model's behavior through Trojaned training samples, which can later be exploited. Guided by basic neuroscientific principles we discover subtle -- yet critical -- structural deviation characterizing Trojaned models. In our analysis we use topological tools. They allow us to model high-order dependencies in the networks, robustly compare different networks, and localize structural abnormalities. One interesting observation is that Trojaned models develop short-cuts from input to output layers. Inspired by these observations, we devise a strategy for robust detection of Trojaned models. Compared to standard baselines it displays better performance on multiple benchmarks.