# Sleeper Agent: Scalable Hidden Trigger Backdoors for Neural Networks Trained from Scratch

arXiv: https://arxiv.org/abs/2106.08970 · HTML: https://mlsanity.com/paper/2106.08970/

## Key facts

- Cited by 102 papers in the map (data through 14 Oct 2026).
- Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security.
- Builds on 35 of its 63 references that are in the map.
- First posted on arXiv 16 Jun 2021.

## Placed in

- LLM safety and security / [Backdoor and data poisoning attacks](https://mlsanity.com/safety/backdoor-and-data-poisoning-attacks/)

## Builds on (most cited first, 30 of 35)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain](https://mlsanity.com/paper/1708.06733/) (2017)
- [Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning](https://mlsanity.com/paper/1712.05526/) (2017)
- [How To Backdoor Federated Learning](https://mlsanity.com/paper/1807.00459/) (2018)
- [Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks](https://mlsanity.com/paper/1804.00792/) (2018)
- [Dataset Condensation with Gradient Matching](https://mlsanity.com/paper/2006.05929/) (2020)
- [Spectral Signatures in Backdoor Attacks](https://mlsanity.com/paper/1811.00636/) (2018)
- [STRIP: A Defence Against Trojan Attacks on Deep Neural Networks](https://mlsanity.com/paper/1902.06531/) (2019)
- [Detecting Backdoor Attacks on Deep Neural Networks by Activation Clustering](https://mlsanity.com/paper/1811.03728/) (2018)
- [Backdoor Learning: A Survey](https://mlsanity.com/paper/2007.08745/) (2020)
- [Hidden Trigger Backdoor Attacks](https://mlsanity.com/paper/1910.00033/) (2019)
- [Invisible Backdoor Attack with Sample-Specific Triggers](https://mlsanity.com/paper/2012.03816/) (2020)
- [Reflection Backdoor: A Natural Backdoor Attack on Deep Neural Networks](https://mlsanity.com/paper/2007.02343/) (2020)
- [Certified Defenses for Data Poisoning Attacks](https://mlsanity.com/paper/1706.03691/) (2017)
- [Label-Consistent Backdoor Attacks](https://mlsanity.com/paper/1912.02771/) (2019)
- [Embedding Watermarks into Deep Neural Networks](https://mlsanity.com/paper/1701.04082/) (2017)
- [A new Backdoor Attack in CNNs by training set corruption without label poisoning](https://mlsanity.com/paper/1902.11237/) (2019)
- [Towards Poisoning of Deep Learning Algorithms with Back-gradient Optimization](https://mlsanity.com/paper/1708.08689/) (2017)
- [Unlearnable Examples: Making Personal Data Unexploitable](https://mlsanity.com/paper/2101.04898/) (2021)
- [Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defenses](https://mlsanity.com/paper/2012.10544/) (2020)
- [Witches' Brew: Industrial Scale Data Poisoning via Gradient Matching](https://mlsanity.com/paper/2009.02276/) (2020)
- [Backdoor Attacks Against Deep Learning Systems in the Physical World](https://mlsanity.com/paper/2006.14580/) (2020)
- [Transferable Clean-Label Poisoning Attacks on Deep Neural Nets](https://mlsanity.com/paper/1905.05897/) (2019)
- [MetaPoison: Practical General-purpose Clean-label Data Poisoning](https://mlsanity.com/paper/2004.00225/) (2020)
- [Just How Toxic is Data Poisoning? A Unified Benchmark for Backdoor and Data Poisoning Attacks](https://mlsanity.com/paper/2006.12557/) (2020)

## Built on by (most cited first, 30 of 102)

- [Poisoning Web-Scale Training Datasets is Practical](https://mlsanity.com/paper/2302.10149/) (2023)
- [BackdoorBench: A Comprehensive Benchmark of Backdoor Learning](https://mlsanity.com/paper/2206.12654/) (2022)
- [Adversarial Examples Make Strong Poisons](https://mlsanity.com/paper/2106.10807/) (2021)
- [Narcissus: A Practical Clean-Label Backdoor Attack with Limited Information](https://mlsanity.com/paper/2204.05255/) (2022)
- [Wild Patterns Reloaded: A Survey of Machine Learning Security against Training Data Poisoning](https://mlsanity.com/paper/2205.01992/) (2022)
- [Untargeted Backdoor Watermark: Towards Harmless and Stealthy Dataset Copyright Protection](https://mlsanity.com/paper/2210.00875/) (2022)
- [Domain Watermark: Effective and Harmless Dataset Copyright Protection is Closed at Hand](https://mlsanity.com/paper/2310.14942/) (2023)
- [Detecting Backdoors During the Inference Stage Based on Corruption Robustness Consistency](https://mlsanity.com/paper/2303.18191/) (2023)
- [Black-box Backdoor Defense via Zero-shot Image Purification](https://mlsanity.com/paper/2303.12175/) (2023)
- [Shared Adversarial Unlearning: Backdoor Mitigation by Unlearning Shared Adversarial Examples](https://mlsanity.com/paper/2307.10562/) (2023)
- [Attention-Enhancing Backdoor Attacks Against BERT-based Models](https://mlsanity.com/paper/2310.14480/) (2023)
- [Mitigating Fine-tuning based Jailbreak Attack with Backdoor Enhanced Safety Alignment](https://mlsanity.com/paper/2402.14968/) (2024)
- [Not All Poisons are Created Equal: Robust Training against Data Poisoning](https://mlsanity.com/paper/2210.09671/) (2022)
- [Towards Stealthy Backdoor Attacks against Speech Recognition via Elements of Sound](https://mlsanity.com/paper/2307.08208/) (2023)
- [ASSET: Robust Backdoor Data Detection Across a Multiplicity of Deep Learning Paradigms](https://mlsanity.com/paper/2302.11408/) (2023)
- [BackdoorBox: A Python Toolbox for Backdoor Learning](https://mlsanity.com/paper/2302.01762/) (2023)
- [Stable Unlearnable Example: Enhancing the Robustness of Unlearnable Examples via Stable Error-Minimizing Noise](https://mlsanity.com/paper/2311.13091/) (2023)
- [Hardly Perceptible Trojan Attack against Neural Networks with Bit Flips](https://mlsanity.com/paper/2207.13417/) (2022)
- [Friendly Noise against Adversarial Noise: A Powerful Defense against Data Poisoning Attacks](https://mlsanity.com/paper/2208.10224/) (2022)
- [Attacks in Adversarial Machine Learning: A Systematic Survey from the Life-cycle Perspective](https://mlsanity.com/paper/2302.09457/) (2023)
- [BackdoorBench: A Comprehensive Benchmark and Analysis of Backdoor Learning](https://mlsanity.com/paper/2407.19845/) (2024)
- [Plug-In Inversion: Model-Agnostic Inversion for Vision with Data Augmentations](https://mlsanity.com/paper/2201.12961/) (2022)
- [Natural Reflection Backdoor Attack on Vision Language Model for Autonomous Driving](https://mlsanity.com/paper/2505.06413/) (2025)
- [On the Effectiveness of Adversarial Training against Backdoor Attacks](https://mlsanity.com/paper/2202.10627/) (2022)
- [Mitigating Backdoor Attack by Injecting Proactive Defensive Backdoor](https://mlsanity.com/paper/2405.16112/) (2024)
- [Generalization Bound and New Algorithm for Clean-Label Backdoor Attack](https://mlsanity.com/paper/2406.00588/) (2024)
- [Imperceptible and Multi-channel Backdoor Attack against Deep Neural Networks](https://mlsanity.com/paper/2201.13164/) (2022)
- [Temporal Robustness against Data Poisoning](https://mlsanity.com/paper/2302.03684/) (2023)
- [Sharpness-Aware Data Poisoning Attack](https://mlsanity.com/paper/2305.14851/) (2023)
- [Demystifying Poisoning Backdoor Attacks from a Statistical Perspective](https://mlsanity.com/paper/2310.10780/) (2023)

## Abstract (arXiv)

As the curation of data for machine learning becomes increasingly automated, dataset tampering is a mounting threat. Backdoor attackers tamper with training data to embed a vulnerability in models that are trained on that data. This vulnerability is then activated at inference time by placing a "trigger" into the model's input. Typical backdoor attacks insert the trigger directly into the training data, although the presence of such an attack may be visible upon inspection. In contrast, the Hidden Trigger Backdoor Attack achieves poisoning without placing a trigger into the training data at all. However, this hidden trigger attack is ineffective at poisoning neural networks trained from scratch. We develop a new hidden trigger attack, Sleeper Agent, which employs gradient matching, data selection, and target model re-training during the crafting process. Sleeper Agent is the first hidden trigger backdoor attack to be effective against neural networks trained from scratch. We demonstrate its effectiveness on ImageNet and in black-box settings. Our implementation code can be found at https://github.com/hsouri/Sleeper-Agent.