# The TrojAI Software Framework: An OpenSource tool for Embedding Trojans into Deep Learning Models

arXiv: https://arxiv.org/abs/2003.07233 · HTML: https://mlsanity.com/paper/2003.07233/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Mar 2020.

## Built on by (most cited first, 30 of 42)

- [Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training](https://mlsanity.com/paper/2401.05566/) (2024)
- [Unsolved Problems in ML Safety](https://mlsanity.com/paper/2109.13916/) (2021)
- [BackdoorBench: A Comprehensive Benchmark of Backdoor Learning](https://mlsanity.com/paper/2206.12654/) (2022)
- [Training with More Confidence: Mitigating Injected and Natural Backdoors During Training](https://mlsanity.com/paper/2202.06382/) (2022)
- [Dual-Key Multimodal Backdoors for Visual Question Answering](https://mlsanity.com/paper/2112.07668/) (2021)
- [Poisoning Deep Reinforcement Learning Agents with In-Distribution Triggers](https://mlsanity.com/paper/2106.07798/) (2021)
- [BackdoorBench: A Comprehensive Benchmark and Analysis of Backdoor Learning](https://mlsanity.com/paper/2407.19845/) (2024)
- [Provable Defense against Backdoor Policies in Reinforcement Learning](https://mlsanity.com/paper/2211.10530/) (2022)
- [Trojan Signatures in DNN Weights](https://mlsanity.com/paper/2109.02836/) (2021)
- [TOP: Backdoor Detection in Neural Networks via Transferability of Perturbation](https://mlsanity.com/paper/2103.10274/) (2021)
- [Learning to Interpret Weight Differences in Language Models](https://mlsanity.com/paper/2510.05092/) (2025)
- [TAD: Trigger Approximation based Black-box Trojan Detection for AI](https://mlsanity.com/paper/2102.01815/) (2021)
- [A Holistic Assessment of the Reliability of Machine Learning Systems](https://mlsanity.com/paper/2307.10586/) (2023)
- [Rethinking Backdoor Detection Evaluation for Language Models](https://mlsanity.com/paper/2409.00399/) (2024)
- [BackdoorDM: A Comprehensive Benchmark for Backdoor Learning on Diffusion Model](https://mlsanity.com/paper/2502.11798/) (2025)
- [MISA: Online Defense of Trojaned Models using Misattributions](https://mlsanity.com/paper/2103.15918/) (2021)
- [PerD: Perturbation Sensitivity-based Neural Trojan Detection Framework on NLP Applications](https://mlsanity.com/paper/2208.04943/) (2022)
- [Gradient Shaping: Enhancing Backdoor Attack Against Reverse Engineering](https://mlsanity.com/paper/2301.12318/) (2023)
- [Backdoor Learning for NLP: Recent Advances, Challenges, and Future Research Directions](https://mlsanity.com/paper/2302.06801/) (2023)
- [BackdoorMBTI: A Backdoor Learning Multimodal Benchmark Tool Kit for Backdoor Defense Evaluation](https://mlsanity.com/paper/2411.11006/) (2024)
- [Scanning Trojaned Models Using Out-of-Distribution Samples](https://mlsanity.com/paper/2501.17151/) (2025)
- [MLDS: A Dataset for Weight-Space Analysis of Neural Networks](https://mlsanity.com/paper/2104.10555/) (2021)
- [Self-supervised Adaptive Pre-training of Multilingual Speech Models for Language and Dialect Identification](https://mlsanity.com/paper/2312.07338/) (2023)
- [Revisiting Training-Inference Trigger Intensity in Backdoor Attacks](https://mlsanity.com/paper/2503.12058/) (2025)
- [SanitAIs: Unsupervised Data Augmentation to Sanitize Trojaned Neural Networks](https://mlsanity.com/paper/2109.04566/) (2021)
- [An Adaptive Black-box Backdoor Detection Method for Deep Neural Networks](https://mlsanity.com/paper/2204.04329/) (2022)
- [Solving Trojan Detection Competitions with Linear Weight Classification](https://mlsanity.com/paper/2411.03445/) (2024)
- [BackFed: An Efficient & Standardized Benchmark Suite for Backdoor Attacks in Federated Learning](https://mlsanity.com/paper/2507.04903/) (2025)
- [DISTIL: Data-Free Inversion of Suspicious Trojan Inputs via Latent Diffusion](https://mlsanity.com/paper/2507.22813/) (2025)
- [Signal Transformer: Complex-valued Attention and Meta-Learning for Signal Recognition](https://mlsanity.com/paper/2106.04392/) (2021)

## Abstract (arXiv)

In this paper, we introduce the TrojAI software framework, an open source set of Python tools capable of generating triggered (poisoned) datasets and associated deep learning (DL) models with trojans at scale. We utilize the developed framework to generate a large set of trojaned MNIST classifiers, as well as demonstrate the capability to produce a trojaned reinforcement-learning model using vector observations. Results on MNIST show that the nature of the trigger, training batch size, and dataset poisoning percentage all affect successful embedding of trojans. We test Neural Cleanse against the trojaned MNIST models and successfully detect anomalies in the trained models approximately 18% of the time. Our experiments and workflow indicate that the TrojAI software framework will enable researchers to easily understand the effects of various configurations of the dataset and training hyperparameters on the generated trojaned deep learning model, and can be used to rapidly and comprehensively test new trojan detection methods.