# DeepPayload: Black-box Backdoor Attack on Deep Learning Models through Neural Payload Injection

arXiv: https://arxiv.org/abs/2101.06896 · HTML: https://mlsanity.com/paper/2101.06896/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 58 references that are in the map.
- First posted on arXiv 18 Jan 2021.

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain](https://mlsanity.com/paper/1708.06733/) (2017)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning](https://mlsanity.com/paper/1712.05526/) (2017)
- [Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks](https://mlsanity.com/paper/1805.12185/) (2018)
- [AMC: AutoML for Model Compression and Acceleration on Mobile Devices](https://mlsanity.com/paper/1802.03494/) (2018)
- [DeepXplore: Automated Whitebox Testing of Deep Learning Systems](https://mlsanity.com/paper/1705.06640/) (2017)
- [DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars](https://mlsanity.com/paper/1708.08559/) (2017)
- [Neural Trojans](https://mlsanity.com/paper/1710.00942/) (2017)
- [DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems](https://mlsanity.com/paper/1803.07519/) (2018)
- [Deep Learning in Mobile and Wireless Networking: A Survey](https://mlsanity.com/paper/1803.04311/) (2018)
- [Slalom: Fast, Verifiable and Private Execution of Neural Networks in Trusted Hardware](https://mlsanity.com/paper/1806.03287/) (2018)
- [A First Look at Deep Learning Apps on Smartphones](https://mlsanity.com/paper/1812.05448/) (2018)
- [ReluDiff: Differential Verification of Deep Neural Networks](https://mlsanity.com/paper/2001.03662/) (2020)
- [daBNN: A Super Fast Inference Framework for Binary Neural Networks on ARM devices](https://mlsanity.com/paper/1908.05858/) (2019)
- [Robustness of on-device Models: Adversarial Attack to Deep Learning Models on Android Apps](https://mlsanity.com/paper/2101.04401/) (2021)
- [Dynamic Slicing for Deep Neural Networks](https://mlsanity.com/paper/2009.13747/) (2020)
- [Sparsifying Neural Network Connections for Face Recognition](https://mlsanity.com/paper/1512.01891/) (2015)
- [Automated Extraction of Personal Knowledge from Smartphone Push Notifications](https://mlsanity.com/paper/1808.02013/) (2018)

## Built on by (most cited first, 30 of 44)

- [An Overview of Backdoor Attacks Against Deep Neural Networks and Possible Defences](https://mlsanity.com/paper/2111.08429/) (2021)
- [NeuronFair: Interpretable White-Box Fairness Testing through Biased Neuron Identification](https://mlsanity.com/paper/2112.13214/) (2021)
- [Test-Time Backdoor Attacks on Multimodal Large Language Models](https://mlsanity.com/paper/2402.08577/) (2024)
- [Towards Practical Deployment-Stage Backdoor Attack on Deep Neural Networks](https://mlsanity.com/paper/2111.12965/) (2021)
- [ModelDiff: Testing-Based DNN Similarity Comparison for Model Reuse Detection](https://mlsanity.com/paper/2106.08890/) (2021)
- [Marksman Backdoor: Backdoor Attacks with Arbitrary Target Class](https://mlsanity.com/paper/2210.09194/) (2022)
- [Label Poisoning is All You Need](https://mlsanity.com/paper/2310.18933/) (2023)
- [Architectural Backdoors in Neural Networks](https://mlsanity.com/paper/2206.07840/) (2022)
- [Backdoor Federated Learning by Poisoning Backdoor-Critical Layers](https://mlsanity.com/paper/2308.04466/) (2023)
- [Data and Model Poisoning Backdoor Attacks on Wireless Federated Learning, and the Defense Mechanisms: A Comprehensive Survey](https://mlsanity.com/paper/2312.08667/) (2023)
- [A Survey of Neural Trojan Attacks and Defenses in Deep Learning](https://mlsanity.com/paper/2202.07183/) (2022)
- [A Large-Scale Exploit Instrumentation Study of AI/ML Supply Chain Attacks in Hugging Face Models](https://mlsanity.com/paper/2410.04490/) (2024)
- [Smart App Attack: Hacking Deep Learning Models in Android Apps](https://mlsanity.com/paper/2204.11075/) (2022)
- [TIJO: Trigger Inversion with Joint Optimization for Defending Multimodal Backdoored Models](https://mlsanity.com/paper/2308.03906/) (2023)
- [Beating Backdoor Attack at Its Own Game](https://mlsanity.com/paper/2307.15539/) (2023)
- [Backdoor Attacks on Time Series: A Generative Approach](https://mlsanity.com/paper/2211.07915/) (2022)
- [ImpNet: Imperceptible and blackbox-undetectable backdoors in compiled neural networks](https://mlsanity.com/paper/2210.00108/) (2022)
- [DataElixir: Purifying Poisoned Dataset to Mitigate Backdoor Attacks via Diffusion Models](https://mlsanity.com/paper/2312.11057/) (2023)
- [Efficient Backdoor Attacks for Deep Neural Networks in Real-world Scenarios](https://mlsanity.com/paper/2306.08386/) (2023)
- [Influencer Backdoor Attack on Semantic Segmentation](https://mlsanity.com/paper/2303.12054/) (2023)
- [IPA-NeRF: Illusory Poisoning Attack Against Neural Radiance Fields](https://mlsanity.com/paper/2407.11921/) (2024)
- [Backdoor Learning for NLP: Recent Advances, Challenges, and Future Research Directions](https://mlsanity.com/paper/2302.06801/) (2023)
- [CertPri: Certifiable Prioritization for Deep Neural Networks via Movement Cost in Feature Space](https://mlsanity.com/paper/2307.09375/) (2023)
- [Graph Neural Backdoor: Fundamentals, Methodologies, Applications, and Future Directions](https://mlsanity.com/paper/2406.10573/) (2024)
- [Hide and Seek: on the Stealthiness of Attacks against Deep Learning Systems](https://mlsanity.com/paper/2205.15944/) (2022)
- [PatchBackdoor: Backdoor Attack against Deep Neural Networks without Model Modification](https://mlsanity.com/paper/2308.11822/) (2023)
- [Towards Secure MLOps: Surveying Attacks, Mitigation Strategies, and Research Challenges](https://mlsanity.com/paper/2506.02032/) (2025)
- [Beyond the Model: Data Pre-processing Attack to Deep Learning Models in Android Apps](https://mlsanity.com/paper/2305.03963/) (2023)
- [Lazy Layers to Make Fine-Tuned Diffusion Models More Traceable](https://mlsanity.com/paper/2405.00466/) (2024)
- [Revisiting Training-Inference Trigger Intensity in Backdoor Attacks](https://mlsanity.com/paper/2503.12058/) (2025)

## Abstract (arXiv)

Deep learning models are increasingly used in mobile applications as critical components. Unlike the program bytecode whose vulnerabilities and threats have been widely-discussed, whether and how the deep learning models deployed in the applications can be compromised are not well-understood since neural networks are usually viewed as a black box. In this paper, we introduce a highly practical backdoor attack achieved with a set of reverse-engineering techniques over compiled deep learning models. The core of the attack is a neural conditional branch constructed with a trigger detector and several operators and injected into the victim model as a malicious payload. The attack is effective as the conditional logic can be flexibly customized by the attacker, and scalable as it does not require any prior knowledge from the original model. We evaluated the attack effectiveness using 5 state-of-the-art deep learning models and real-world samples collected from 30 users. The results demonstrated that the injected backdoor can be triggered with a success rate of 93.5%, while only brought less than 2ms latency overhead and no more than 1.4% accuracy decrease. We further conducted an empirical study on real-world mobile deep learning apps collected from Google Play. We found 54 apps that were vulnerable to our attack, including popular and security-critical ones. The results call for the awareness of deep learning application developers and auditors to enhance the protection of deployed models.