# Subnet Replacement: Deployment-stage backdoor attack against deep neural networks in gray-box setting

arXiv: https://arxiv.org/abs/2107.07240 · HTML: https://mlsanity.com/paper/2107.07240/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 24 references that are in the map.
- First posted on arXiv 15 Jul 2021.

## Builds on (most cited first, 11 of 11)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning](https://mlsanity.com/paper/1712.05526/) (2017)
- [Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks](https://mlsanity.com/paper/1805.12185/) (2018)
- [Hidden Trigger Backdoor Attacks](https://mlsanity.com/paper/1910.00033/) (2019)
- [Weight Poisoning Attacks on Pre-trained Models](https://mlsanity.com/paper/2004.06660/) (2020)
- [Certified Defenses for Data Poisoning Attacks](https://mlsanity.com/paper/1706.03691/) (2017)
- [Detecting AI Trojans Using Meta Neural Analysis](https://mlsanity.com/paper/1910.03137/) (2019)
- [Bit-Flip Attack: Crushing Neural Network with Progressive Bit Search](https://mlsanity.com/paper/1903.12269/) (2019)
- [Defending Against Physically Realizable Attacks on Image Classification](https://mlsanity.com/paper/1909.09552/) (2019)
- [Targeted Attack against Deep Neural Networks via Flipping Limited Weight Bits](https://mlsanity.com/paper/2102.10496/) (2021)
- [Detection of Adversarial Training Examples in Poisoning Attacks through Anomaly Detection](https://mlsanity.com/paper/1802.03041/) (2018)

## Built on by (most cited first, 21 of 21)

- [Narcissus: A Practical Clean-Label Backdoor Attack with Limited Information](https://mlsanity.com/paper/2204.05255/) (2022)
- [BadChain: Backdoor Chain-of-Thought Prompting for Large Language Models](https://mlsanity.com/paper/2401.12242/) (2024)
- [Towards A Proactive ML Approach for Detecting Backdoor Poison Samples](https://mlsanity.com/paper/2205.13616/) (2022)
- [Attacks in Adversarial Machine Learning: A Systematic Survey from the Life-cycle Perspective](https://mlsanity.com/paper/2302.09457/) (2023)
- [Discovering Knowledge-Critical Subnetworks in Pretrained Language Models](https://mlsanity.com/paper/2310.03084/) (2023)
- [BaDExpert: Extracting Backdoor Functionality for Accurate Backdoor Input Detection](https://mlsanity.com/paper/2308.12439/) (2023)
- [Backdoor Attacks on Time Series: A Generative Approach](https://mlsanity.com/paper/2211.07915/) (2022)
- [ImpNet: Imperceptible and blackbox-undetectable backdoors in compiled neural networks](https://mlsanity.com/paper/2210.00108/) (2022)
- [Security Considerations in AI-Robotics: A Survey of Current Methods, Challenges, and Opportunities](https://mlsanity.com/paper/2310.08565/) (2023)
- [On Feasibility of Server-side Backdoor Attacks on Split Learning](https://mlsanity.com/paper/2302.09578/) (2023)
- [GaussTrap: Stealthy Poisoning Attacks on 3D Gaussian Splatting for Targeted Scene Confusion](https://mlsanity.com/paper/2504.20829/) (2025)
- [Influencer Backdoor Attack on Semantic Segmentation](https://mlsanity.com/paper/2303.12054/) (2023)
- [IPA-NeRF: Illusory Poisoning Attack Against Neural Radiance Fields](https://mlsanity.com/paper/2407.11921/) (2024)
- [Backdoors Stuck At The Frontdoor: Multi-Agent Backdoor Attacks That Backfire](https://mlsanity.com/paper/2201.12211/) (2022)
- [On the Validity of Traditional Vulnerability Scoring Systems for Adversarial Attacks against LLMs](https://mlsanity.com/paper/2412.20087/) (2024)
- [Revisiting Training-Inference Trigger Intensity in Backdoor Attacks](https://mlsanity.com/paper/2503.12058/) (2025)
- [Circumventing Backdoor Defenses That Are Based on Latent Separability](https://mlsanity.com/paper/2205.13613/) (2022)
- [BadCS: A Backdoor Attack Framework for Code search](https://mlsanity.com/paper/2305.05503/) (2023)
- [Large Language Models Can Verbatim Reproduce Long Malicious Sequences](https://mlsanity.com/paper/2503.17578/) (2025)
- [Can We Rely on AI?](https://mlsanity.com/paper/2308.15092/) (2023)
- [Backdoor Attacks on Deep Learning Face Detection](https://mlsanity.com/paper/2508.00620/) (2025)

## Abstract (arXiv)

We study the realistic potential of conducting backdoor attack against deep neural networks (DNNs) during deployment stage. Specifically, our goal is to design a deployment-stage backdoor attack algorithm that is both threatening and realistically implementable. To this end, we propose Subnet Replacement Attack (SRA), which is capable of embedding backdoor into DNNs by directly modifying a limited number of model parameters. Considering the realistic practicability, we abandon the strong white-box assumption widely adopted in existing studies, instead, our algorithm works in a gray-box setting, where architecture information of the victim model is available but the adversaries do not have any knowledge of parameter values. The key philosophy underlying our approach is -- given any neural network instance (regardless of its specific parameter values) of a certain architecture, we can always embed a backdoor into that model instance, by replacing a very narrow subnet of a benign model (without backdoor) with a malicious backdoor subnet, which is designed to be sensitive (fire large activation value) to a particular backdoor trigger pattern.