# Detecting Trojaned DNNs Using Counterfactual Attributions

arXiv: https://arxiv.org/abs/2012.02275 · HTML: https://mlsanity.com/paper/2012.02275/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Dec 2020.

## Built on by (most cited first, 4 of 4)

- [Trigger Hunting with a Topological Prior for Trojan Detection](https://mlsanity.com/paper/2110.08335/) (2021)
- [TIJO: Trigger Inversion with Joint Optimization for Defending Multimodal Backdoored Models](https://mlsanity.com/paper/2308.03906/) (2023)
- [COLLIDER: A Robust Training Framework for Backdoor Data](https://mlsanity.com/paper/2210.06704/) (2022)
- [EX-RAY: Distinguishing Injected Backdoor from Natural Features in Neural Networks by Examining Differential Feature Symmetry](https://mlsanity.com/paper/2103.08820/) (2021)

## Abstract (arXiv)

We target the problem of detecting Trojans or backdoors in DNNs. Such models behave normally with typical inputs but produce specific incorrect predictions for inputs poisoned with a Trojan trigger. Our approach is based on a novel observation that the trigger behavior depends on a few ghost neurons that activate on trigger pattern and exhibit abnormally higher relative attribution for wrong decisions when activated. Further, these trigger neurons are also active on normal inputs of the target class. Thus, we use counterfactual attributions to localize these ghost neurons from clean inputs and then incrementally excite them to observe changes in the model's accuracy. We use this information for Trojan detection by using a deep set encoder that enables invariance to the number of model classes, architecture, etc. Our approach is implemented in the TrinityAI tool that exploits the synergies between trustworthiness, resilience, and interpretability challenges in deep learning. We evaluate our approach on benchmarks with high diversity in model architectures, triggers, etc. We show consistent gains (+10%) over state-of-the-art methods that rely on the susceptibility of the DNN to specific adversarial attacks, which in turn requires strong assumptions on the nature of the Trojan attack.