# Demon in the Variant: Statistical Analysis of DNNs for Robust Backdoor Contamination Detection

arXiv: https://arxiv.org/abs/1908.00686 · HTML: https://mlsanity.com/paper/1908.00686/

## Key facts

- Cited by 124 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Aug 2019.

## Built on by (most cited first, 30 of 124)

- [Anti-Backdoor Learning: Training Clean Models on Poisoned Data](https://mlsanity.com/paper/2110.11571/) (2021)
- [Dataset Distillation: A Comprehensive Review](https://mlsanity.com/paper/2301.07014/) (2023)
- [BadEncoder: Backdoor Attacks to Pre-trained Encoders in Self-Supervised Learning](https://mlsanity.com/paper/2108.00352/) (2021)
- [Wild Patterns Reloaded: A Survey of Machine Learning Security against Training Data Poisoning](https://mlsanity.com/paper/2205.01992/) (2022)
- [Hidden Backdoors in Human-Centric Language Models](https://mlsanity.com/paper/2105.00164/) (2021)
- [Handcrafted Backdoors in Deep Neural Networks](https://mlsanity.com/paper/2106.04690/) (2021)
- [The "Beatrix'' Resurrections: Robust Backdoor Detection via Gram Matrices](https://mlsanity.com/paper/2209.11715/) (2022)
- [Reconstructive Neuron Pruning for Backdoor Defense](https://mlsanity.com/paper/2305.14876/) (2023)
- ["Real Attackers Don't Compute Gradients": Bridging the Gap Between Adversarial ML Research and Practice](https://mlsanity.com/paper/2212.14315/) (2022)
- [Measuring the Effect of Training Data on Deep Learning Predictions via Randomized Experiments](https://mlsanity.com/paper/2206.10013/) (2022)
- [Stealthy Backdoor Attack for Code Models](https://mlsanity.com/paper/2301.02496/) (2023)
- [Detecting Backdoors During the Inference Stage Based on Corruption Robustness Consistency](https://mlsanity.com/paper/2303.18191/) (2023)
- [UNICORN: A Unified Backdoor Trigger Inversion Framework](https://mlsanity.com/paper/2304.02786/) (2023)
- [An Overview of Backdoor Attacks Against Deep Neural Networks and Possible Defences](https://mlsanity.com/paper/2111.08429/) (2021)
- [Poison Forensics: Traceback of Data Poisoning Attacks in Neural Networks](https://mlsanity.com/paper/2110.06904/) (2021)
- [Towards A Proactive ML Approach for Detecting Backdoor Poison Samples](https://mlsanity.com/paper/2205.13616/) (2022)
- [IBD-PSC: Input-level Backdoor Detection via Parameter-oriented Scaling Consistency](https://mlsanity.com/paper/2405.09786/) (2024)
- [Training with More Confidence: Mitigating Injected and Natural Backdoors During Training](https://mlsanity.com/paper/2202.06382/) (2022)
- [Backdoor Defense via Deconfounded Representation Learning](https://mlsanity.com/paper/2303.06818/) (2023)
- [A Survey of Recent Backdoor Attacks and Defenses in Large Language Models](https://mlsanity.com/paper/2406.06852/) (2024)
- [Towards Practical Deployment-Stage Backdoor Attack on Deep Neural Networks](https://mlsanity.com/paper/2111.12965/) (2021)
- [An Embarrassingly Simple Backdoor Attack on Self-supervised Learning](https://mlsanity.com/paper/2210.07346/) (2022)
- [Elijah: Eliminating Backdoors Injected in Diffusion Models via Distribution Shift](https://mlsanity.com/paper/2312.00050/) (2023)
- [Quantization Backdoors to Deep Learning Commercial Frameworks](https://mlsanity.com/paper/2108.09187/) (2021)
- [ASSET: Robust Backdoor Data Detection Across a Multiplicity of Deep Learning Paradigms](https://mlsanity.com/paper/2302.11408/) (2023)
- [Machine Learning Security against Data Poisoning: Are We There Yet?](https://mlsanity.com/paper/2204.05986/) (2022)
- [Backdoor Attacks Against Dataset Distillation](https://mlsanity.com/paper/2301.01197/) (2023)
- [Chameleon: Adapting to Peer Images for Planting Durable Backdoors in Federated Learning](https://mlsanity.com/paper/2304.12961/) (2023)
- [Distilling Cognitive Backdoor Patterns within an Image](https://mlsanity.com/paper/2301.10908/) (2023)
- [Qu-ANTI-zation: Exploiting Quantization Artifacts for Achieving Adversarial Outcomes](https://mlsanity.com/paper/2110.13541/) (2021)

## Abstract (arXiv)

A security threat to deep neural networks (DNN) is backdoor contamination, in which an adversary poisons the training data of a target model to inject a Trojan so that images carrying a specific trigger will always be classified into a specific label. Prior research on this problem assumes the dominance of the trigger in an image's representation, which causes any image with the trigger to be recognized as a member in the target class. Such a trigger also exhibits unique features in the representation space and can therefore be easily separated from legitimate images. Our research, however, shows that simple target contamination can cause the representation of an attack image to be less distinguishable from that of legitimate ones, thereby evading existing defenses against the backdoor infection. In our research, we show that such a contamination attack actually subtly changes the representation distribution for the target class, which can be captured by a statistic analysis. More specifically, we leverage an EM algorithm to decompose an image into its identity part (e.g., person, traffic sign) and variation part within a class (e.g., lighting, poses). Then we analyze the distribution in each class, identifying those more likely to be characterized by a mixture model resulted from adding attack samples to the legitimate image pool. Our research shows that this new technique effectively detects data contamination attacks, including the new one we propose, and is also robust against the evasion attempts made by a knowledgeable adversary.