# Towards Practical Deployment-Stage Backdoor Attack on Deep Neural Networks

arXiv: https://arxiv.org/abs/2111.12965 · HTML: https://mlsanity.com/paper/2111.12965/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 78 references that are in the map.
- First posted on arXiv 25 Nov 2021.

## Builds on (most cited first, 30 of 44)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Adversarial examples in the physical world](https://mlsanity.com/paper/1607.02533/) (2016)
- [BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain](https://mlsanity.com/paper/1708.06733/) (2017)
- [Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning](https://mlsanity.com/paper/1712.05526/) (2017)
- [Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks](https://mlsanity.com/paper/1805.12185/) (2018)
- [Adversarial Patch](https://mlsanity.com/paper/1712.09665/) (2017)
- [Spectral Signatures in Backdoor Attacks](https://mlsanity.com/paper/1811.00636/) (2018)
- [STRIP: A Defence Against Trojan Attacks on Deep Neural Networks](https://mlsanity.com/paper/1902.06531/) (2019)
- [Detecting Backdoor Attacks on Deep Neural Networks by Activation Clustering](https://mlsanity.com/paper/1811.03728/) (2018)
- [Hidden Trigger Backdoor Attacks](https://mlsanity.com/paper/1910.00033/) (2019)
- [Weight Poisoning Attacks on Pre-trained Models](https://mlsanity.com/paper/2004.06660/) (2020)
- [A backdoor attack against LSTM-based text classification systems](https://mlsanity.com/paper/1905.12457/) (2019)
- [Detecting AI Trojans Using Meta Neural Analysis](https://mlsanity.com/paper/1910.03137/) (2019)
- [Neural Trojans](https://mlsanity.com/paper/1710.00942/) (2017)
- [SentiNet: Detecting Localized Universal Attacks Against Deep Learning Systems](https://mlsanity.com/paper/1812.00292/) (2018)
- [Backdoor Embedding in Convolutional Neural Network Models via Invisible Perturbation](https://mlsanity.com/paper/1808.10307/) (2018)
- [Backdoor Attacks Against Deep Learning Systems in the Physical World](https://mlsanity.com/paper/2006.14580/) (2020)
- [Backdoor Attacks to Graph Neural Networks](https://mlsanity.com/paper/2006.11165/) (2020)
- [Demon in the Variant: Statistical Analysis of DNNs for Robust Backdoor Contamination Detection](https://mlsanity.com/paper/1908.00686/) (2019)
- [TABOR: A Highly Accurate Approach to Inspecting and Restoring Trojan Backdoors in AI Systems](https://mlsanity.com/paper/1908.01763/) (2019)
- [TBT: Targeted Neural Network Attack with Bit Trojan](https://mlsanity.com/paper/1909.05193/) (2019)
- [Bit-Flip Attack: Crushing Neural Network with Progressive Bit Search](https://mlsanity.com/paper/1903.12269/) (2019)
- [An Embarrassingly Simple Approach for Trojan Attack in Deep Neural Networks](https://mlsanity.com/paper/2006.08131/) (2020)

## Built on by (most cited first, 30 of 34)

- [Untargeted Backdoor Watermark: Towards Harmless and Stealthy Dataset Copyright Protection](https://mlsanity.com/paper/2210.00875/) (2022)
- [Physical Adversarial Attack meets Computer Vision: A Decade Survey](https://mlsanity.com/paper/2209.15179/) (2022)
- [Reconstructive Neuron Pruning for Backdoor Defense](https://mlsanity.com/paper/2305.14876/) (2023)
- [MM-BD: Post-Training Detection of Backdoor Attacks with Arbitrary Backdoor Pattern Types Using a Maximum Margin Statistic](https://mlsanity.com/paper/2205.06900/) (2022)
- [Towards A Proactive ML Approach for Detecting Backdoor Poison Samples](https://mlsanity.com/paper/2205.13616/) (2022)
- [Test-Time Backdoor Attacks on Multimodal Large Language Models](https://mlsanity.com/paper/2402.08577/) (2024)
- [IBD-PSC: Input-level Backdoor Detection via Parameter-oriented Scaling Consistency](https://mlsanity.com/paper/2405.09786/) (2024)
- [FedTracker: Furnishing Ownership Verification and Traceability for Federated Learning Model](https://mlsanity.com/paper/2211.07160/) (2022)
- [BackdoorBox: A Python Toolbox for Backdoor Learning](https://mlsanity.com/paper/2302.01762/) (2023)
- [Attacks in Adversarial Machine Learning: A Systematic Survey from the Life-cycle Perspective](https://mlsanity.com/paper/2302.09457/) (2023)
- [Backdoor Federated Learning by Poisoning Backdoor-Critical Layers](https://mlsanity.com/paper/2308.04466/) (2023)
- [AI Risk Management Should Incorporate Both Safety and Security](https://mlsanity.com/paper/2405.19524/) (2024)
- [Trap and Replace: Defending Backdoor Attacks by Trapping Them into an Easy-to-Replace Subnetwork](https://mlsanity.com/paper/2210.06428/) (2022)
- [Backdoor Attacks with Input-unique Triggers in NLP](https://mlsanity.com/paper/2303.14325/) (2023)
- [FedCIP: Federated Client Intellectual Property Protection with Traitor Tracking](https://mlsanity.com/paper/2306.01356/) (2023)
- [BaDExpert: Extracting Backdoor Functionality for Accurate Backdoor Input Detection](https://mlsanity.com/paper/2308.12439/) (2023)
- [Single Image Backdoor Inversion via Robust Smoothed Classifiers](https://mlsanity.com/paper/2303.00215/) (2023)
- [FLTracer: Accurate Poisoning Attack Provenance in Federated Learning](https://mlsanity.com/paper/2310.13424/) (2023)
- [Invisible Backdoor Attack against Self-supervised Learning](https://mlsanity.com/paper/2405.14672/) (2024)
- [Facial Misrecognition Systems: Simple Weight Manipulations Force DNNs to Err Only on Specific Persons](https://mlsanity.com/paper/2301.03118/) (2023)
- [The Victim and The Beneficiary: Exploiting a Poisoned Model to Train a Clean Model on Poisoned Data](https://mlsanity.com/paper/2404.11265/) (2024)
- [Detecting Sleeper Agents in Large Language Models via Semantic Drift Analysis](https://mlsanity.com/paper/2511.15992/) (2025)
- [DABS: Data-Agnostic Backdoor attack at the Server in Federated Learning](https://mlsanity.com/paper/2305.01267/) (2023)
- [THEMIS: Towards Practical Intellectual Property Protection for Post-Deployment On-Device Deep Learning Models](https://mlsanity.com/paper/2503.23748/) (2025)
- [TED++: Submanifold-Aware Backdoor Detection via Layerwise Tubular-Neighbourhood Screening](https://mlsanity.com/paper/2510.14299/) (2025)
- [Circumventing Backdoor Defenses That Are Based on Latent Separability](https://mlsanity.com/paper/2205.13613/) (2022)
- [Flareon: Stealthy any2any Backdoor Injection via Poisoned Augmentation](https://mlsanity.com/paper/2212.09979/) (2022)
- [EventTrojan: Manipulating Non-Intrusive Speech Quality Assessment via Imperceptible Events](https://mlsanity.com/paper/2309.01480/) (2023)
- [Rethinking Pruning for Backdoor Mitigation: An Optimization Perspective](https://mlsanity.com/paper/2405.17746/) (2024)
- [Magnitude-based Neuron Pruning for Backdoor Defens](https://mlsanity.com/paper/2405.17750/) (2024)

## Abstract (arXiv)

One major goal of the AI security community is to securely and reliably produce and deploy deep learning models for real-world applications. To this end, data poisoning based backdoor attacks on deep neural networks (DNNs) in the production stage (or training stage) and corresponding defenses are extensively explored in recent years. Ironically, backdoor attacks in the deployment stage, which can often happen in unprofessional users' devices and are thus arguably far more threatening in real-world scenarios, draw much less attention of the community. We attribute this imbalance of vigilance to the weak practicality of existing deployment-stage backdoor attack algorithms and the insufficiency of real-world attack demonstrations. To fill the blank, in this work, we study the realistic threat of deployment-stage backdoor attacks on DNNs. We base our study on a commonly used deployment-stage attack paradigm -- adversarial weight attack, where adversaries selectively modify model weights to embed backdoor into deployed DNNs. To approach realistic practicality, we propose the first gray-box and physically realizable weights attack algorithm for backdoor injection, namely subnet replacement attack (SRA), which only requires architecture information of the victim model and can support physical triggers in the real world. Extensive experimental simulations and system-level real-world attack demonstrations are conducted. Our results not only suggest the effectiveness and practicality of the proposed attack algorithm, but also reveal the practical risk of a novel type of computer virus that may widely spread and stealthily inject backdoor into DNN models in user devices. By our study, we call for more attention to the vulnerability of DNNs in the deployment stage.