# TrojanZoo: Towards Unified, Holistic, and Practical Evaluation of Neural Backdoors

arXiv: https://arxiv.org/abs/2012.09302 · HTML: https://mlsanity.com/paper/2012.09302/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Dec 2020.

## Built on by (most cited first, 30 of 30)

- [BackdoorBench: A Comprehensive Benchmark of Backdoor Learning](https://mlsanity.com/paper/2206.12654/) (2022)
- [A Unified Evaluation of Textual Backdoor Learning: Frameworks and Benchmarks](https://mlsanity.com/paper/2206.08514/) (2022)
- [The "Beatrix'' Resurrections: Robust Backdoor Detection via Gram Matrices](https://mlsanity.com/paper/2209.11715/) (2022)
- [An Embarrassingly Simple Backdoor Attack on Self-supervised Learning](https://mlsanity.com/paper/2210.07346/) (2022)
- [Quantization Backdoors to Deep Learning Commercial Frameworks](https://mlsanity.com/paper/2108.09187/) (2021)
- [BackdoorBox: A Python Toolbox for Backdoor Learning](https://mlsanity.com/paper/2302.01762/) (2023)
- [Attacks in Adversarial Machine Learning: A Systematic Survey from the Life-cycle Perspective](https://mlsanity.com/paper/2302.09457/) (2023)
- [BackdoorBench: A Comprehensive Benchmark and Analysis of Backdoor Learning](https://mlsanity.com/paper/2407.19845/) (2024)
- [On the Difficulty of Defending Contrastive Learning against Backdoor Attacks](https://mlsanity.com/paper/2312.09057/) (2023)
- [Reliable Model Watermarking: Defending Against Theft without Compromising on Evasion](https://mlsanity.com/paper/2404.13518/) (2024)
- [BAN: Detecting Backdoors Activated by Adversarial Neuron Noise](https://mlsanity.com/paper/2405.19928/) (2024)
- [BackdoorDM: A Comprehensive Benchmark for Backdoor Learning on Diffusion Model](https://mlsanity.com/paper/2502.11798/) (2025)
- [Salient Conditional Diffusion for Defending Against Backdoor Attacks](https://mlsanity.com/paper/2301.13862/) (2023)
- [Towards Backdoor Stealthiness in Model Parameter Space](https://mlsanity.com/paper/2501.05928/) (2025)
- [Your Compiler is Backdooring Your Model: Understanding and Exploiting Compilation Inconsistency Vulnerabilities in Deep Learning Compilers](https://mlsanity.com/paper/2509.11173/) (2025)
- [Watch the Watcher! Backdoor Attacks on Security-Enhancing Diffusion Models](https://mlsanity.com/paper/2406.09669/) (2024)
- [2D-OOB: Attributing Data Contribution Through Joint Valuation Framework](https://mlsanity.com/paper/2408.03572/) (2024)
- [Backdoor for Debias: Mitigating Model Bias with Backdoor Attack-based Artificial Bias](https://mlsanity.com/paper/2303.01504/) (2023)
- [The Art of Deception: Robust Backdoor Attack using Dynamic Stacking of Triggers](https://mlsanity.com/paper/2401.01537/) (2024)
- [BackdoorMBTI: A Backdoor Learning Multimodal Benchmark Tool Kit for Backdoor Defense Evaluation](https://mlsanity.com/paper/2411.11006/) (2024)
- [On the Permanence of Backdoors in Evolving Models](https://mlsanity.com/paper/2206.04677/) (2022)
- [Universal Soldier: Using Universal Adversarial Perturbations for Detecting Backdoor Attacks](https://mlsanity.com/paper/2302.00747/) (2023)
- [Pick your Poison: Undetectability versus Robustness in Data Poisoning Attacks](https://mlsanity.com/paper/2305.09671/) (2023)
- [BackFed: An Efficient & Standardized Benchmark Suite for Backdoor Attacks in Federated Learning](https://mlsanity.com/paper/2507.04903/) (2025)
- [Amulet: a Python Library for Assessing Interactions Among ML Defenses and Risks](https://mlsanity.com/paper/2509.12386/) (2025)
- [Neural Architectural Backdoors](https://mlsanity.com/paper/2210.12179/) (2022)
- [Dormant Neural Trojans](https://mlsanity.com/paper/2211.01808/) (2022)
- [OVLA: Neural Network Ownership Verification using Latent Watermarks](https://mlsanity.com/paper/2306.13215/) (2023)
- [BackdoorBench: A Comprehensive Benchmark and Analysis of Backdoor Learning](https://mlsanity.com/paper/2401.15002/) (2024)
- [Quantization Blindspots: How Model Compression Breaks Backdoor Defenses](https://mlsanity.com/paper/2512.06243/) (2025)

## Abstract (arXiv)

Neural backdoors represent one primary threat to the security of deep learning systems. The intensive research has produced a plethora of backdoor attacks/defenses, resulting in a constant arms race. However, due to the lack of evaluation benchmarks, many critical questions remain under-explored: (i) what are the strengths and limitations of different attacks/defenses? (ii) what are the best practices to operate them? and (iii) how can the existing attacks/defenses be further improved? To bridge this gap, we design and implement TROJANZOO, the first open-source platform for evaluating neural backdoor attacks/defenses in a unified, holistic, and practical manner. Thus far, focusing on the computer vision domain, it has incorporated 8 representative attacks, 14 state-of-the-art defenses, 6 attack performance metrics, 10 defense utility metrics, as well as rich tools for in-depth analysis of the attack-defense interactions. Leveraging TROJANZOO, we conduct a systematic study on the existing attacks/defenses, unveiling their complex design spectrum: both manifest intricate trade-offs among multiple desiderata (e.g., the effectiveness, evasiveness, and transferability of attacks). We further explore improving the existing attacks/defenses, leading to a number of interesting findings: (i) one-pixel triggers often suffice; (ii) training from scratch often outperforms perturbing benign models to craft trojan models; (iii) optimizing triggers and trojan models jointly greatly improves both attack effectiveness and evasiveness; (iv) individual defenses can often be evaded by adaptive attacks; and (v) exploiting model interpretability significantly improves defense robustness. We envision that TROJANZOO will serve as a valuable platform to facilitate future research on neural backdoors.