# NeuronInspect: Detecting Backdoors in Neural Networks via Output Explanations

arXiv: https://arxiv.org/abs/1911.07399 · HTML: https://mlsanity.com/paper/1911.07399/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Nov 2019.

## Built on by (most cited first, 30 of 50)

- [Poisoning Web-Scale Training Datasets is Practical](https://mlsanity.com/paper/2302.10149/) (2023)
- [Be Careful about Poisoned Word Embeddings: Exploring the Vulnerability of the Embedding Layers in NLP Models](https://mlsanity.com/paper/2103.15543/) (2021)
- [Wild Patterns Reloaded: A Survey of Machine Learning Security against Training Data Poisoning](https://mlsanity.com/paper/2205.01992/) (2022)
- [A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validation](https://mlsanity.com/paper/2305.11391/) (2023)
- [Untargeted Backdoor Watermark: Towards Harmless and Stealthy Dataset Copyright Protection](https://mlsanity.com/paper/2210.00875/) (2022)
- [Black-box Detection of Backdoor Attacks with Limited Information and Data](https://mlsanity.com/paper/2103.13127/) (2021)
- [Fine-Tuning Is All You Need to Mitigate Backdoor Attacks](https://mlsanity.com/paper/2212.09067/) (2022)
- [Towards Practical Deployment-Stage Backdoor Attack on Deep Neural Networks](https://mlsanity.com/paper/2111.12965/) (2021)
- [An Embarrassingly Simple Backdoor Attack on Self-supervised Learning](https://mlsanity.com/paper/2210.07346/) (2022)
- [MOVE: Effective and Harmless Ownership Verification via Embedded External Features](https://mlsanity.com/paper/2208.02820/) (2022)
- [Poisoning Deep Reinforcement Learning Agents with In-Distribution Triggers](https://mlsanity.com/paper/2106.07798/) (2021)
- [Defending Against Patch-based Backdoor Attacks on Self-Supervised Learning](https://mlsanity.com/paper/2304.01482/) (2023)
- [Stealthy Backdoors as Compression Artifacts](https://mlsanity.com/paper/2104.15129/) (2021)
- [A Survey of Neural Trojan Attacks and Defenses in Deep Learning](https://mlsanity.com/paper/2202.07183/) (2022)
- [BEAGLE: Forensics of Deep Learning Backdoor Attack for Better Defense](https://mlsanity.com/paper/2301.06241/) (2023)
- [Jigsaw Puzzle: Selective Backdoor Attack to Subvert Malware Classifiers](https://mlsanity.com/paper/2202.05470/) (2022)
- [Preference Poisoning Attacks on Reward Model Learning](https://mlsanity.com/paper/2402.01920/) (2024)
- [LOTUS: Evasive and Resilient Backdoor Attacks through Sub-Partitioning](https://mlsanity.com/paper/2403.17188/) (2024)
- [TIJO: Trigger Inversion with Joint Optimization for Defending Multimodal Backdoored Models](https://mlsanity.com/paper/2308.03906/) (2023)
- [Generalization Bound and New Algorithm for Clean-Label Backdoor Attack](https://mlsanity.com/paper/2406.00588/) (2024)
- [On the Difficulty of Defending Contrastive Learning against Backdoor Attacks](https://mlsanity.com/paper/2312.09057/) (2023)
- [XGBD: Explanation-Guided Graph Backdoor Detection](https://mlsanity.com/paper/2308.04406/) (2023)
- [EX-RAY: Distinguishing Injected Backdoor from Natural Features in Neural Networks by Examining Differential Feature Symmetry](https://mlsanity.com/paper/2103.08820/) (2021)
- [Towards a Defense Against Federated Backdoor Attacks Under Continuous Training](https://mlsanity.com/paper/2205.11736/) (2022)
- [Not All Prompts Are Secure: A Switchable Backdoor Attack Against Pre-trained Vision Transformers](https://mlsanity.com/paper/2405.10612/) (2024)
- [MISA: Online Defense of Trojaned Models using Misattributions](https://mlsanity.com/paper/2103.15918/) (2021)
- [Backdooring Explainable Machine Learning](https://mlsanity.com/paper/2204.09498/) (2022)
- [Be Careful with Rotation: A Uniform Backdoor Pattern for 3D Shape](https://mlsanity.com/paper/2211.16192/) (2022)
- [SecurityNet: Assessing Machine Learning Vulnerabilities on Public Models](https://mlsanity.com/paper/2310.12665/) (2023)
- [Towards Sample-specific Backdoor Attack with Clean Labels via Attribute Trigger](https://mlsanity.com/paper/2312.04584/) (2023)

## Abstract (arXiv)

Deep neural networks have achieved state-of-the-art performance on various tasks. However, lack of interpretability and transparency makes it easier for malicious attackers to inject trojan backdoor into the neural networks, which will make the model behave abnormally when a backdoor sample with a specific trigger is input. In this paper, we propose NeuronInspect, a framework to detect trojan backdoors in deep neural networks via output explanation techniques. NeuronInspect first identifies the existence of backdoor attack targets by generating the explanation heatmap of the output layer. We observe that generated heatmaps from clean and backdoored models have different characteristics. Therefore we extract features that measure the attributes of explanations from an attacked model namely: sparse, smooth and persistent. We combine these features and use outlier detection to figure out the outliers, which is the set of attack targets. We demonstrate the effectiveness and efficiency of NeuronInspect on MNIST digit recognition dataset and GTSRB traffic sign recognition dataset. We extensively evaluate NeuronInspect on different attack scenarios and prove better robustness and effectiveness over state-of-the-art trojan backdoor detection techniques Neural Cleanse by a great margin.