mlsanityData to 7 Oct 2026

BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain

arXiv 1708.06733 ↗First posted 22 Aug 2017cs.CR cs.LGcited by 1,052

Not on the map. Why?

Cited by 387 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Deep learning-based techniques have achieved state-of-the-art performance on a wide variety of recognition and classification tasks. However, these networks are typically computationally expensive to train, requiring weeks of computation on many GPUs; as a result, many users outsource the training procedure to the cloud or rely on pre-trained models that are then fine-tuned for a specific task. In this paper we show that outsourced training introduces new security risks: an adversary can create a maliciously trained network (a backdoored neural network, or a BadNet) that has state-of-the-art performance on the user's training and validation samples, but behaves badly on specific attacker-chosen inputs. We first explore the properties of BadNets in a toy example, by creating a backdoored handwritten digit classifier. Next, we demonstrate backdoors in a more realistic scenario by creating a U.S. street sign classifier that identifies stop signs as speed limits when a special sticker is added to the stop sign; we then show in addition that the backdoor in our US street sign detector can persist even if the network is later retrained for another task and cause a drop in accuracy of 25% on average when the backdoor trigger is present. These results demonstrate that backdoors in neural networks are both powerful and---because the behavior of neural networks is difficult to explicate---stealthy. This work provides motivation for further research into techniques for verifying and inspecting neural networks, just as we have developed tools for verifying and debugging software.

Key facts. Cited by 1,052 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Aug 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 133; 2022: 161; 2023: 177; 2024: 241; 2025: 177; 2026: 163.

1,052 papers.

Sort:

By area:

  1. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  2. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  3. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 343
  4. 2021Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networkscited by 313
  5. 2021Unsolved Problems in ML Safetycited by 291
  6. 2023Poisoning Web-Scale Training Datasets is Practicalcited by 276
  7. 2023TRAK: Attributing Model Behavior at Scalecited by 273
  8. 2021Anti-Backdoor Learning: Training Clean Models on Poisoned Datacited by 258
  9. 2024PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Modelscited by 251
  10. 2023Model Sparsity Can Simplify Machine Unlearningcited by 206
  11. 2021Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Triggercited by 203
  12. 2022Datamodels: Predicting Predictions from Training Datacited by 186
38 more
  1. 2021Adversarial Unlearning of Backdoors via Implicit Hypergradientcited by 160
  2. 2023Heterogeneous Federated Learning: State-of-the-art and Research Challengescited by 160
  3. 2021Rethinking the Backdoor Attacks' Triggers: A Frequency Perspectivecited by 155
  4. 2021Poisoning and Backdooring Contrastive Learningcited by 149
  5. 2021Mind the Style of Text! Adversarial and Backdoor Attacks Based on Text Style Transfercited by 147
  6. 2024Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agentscited by 135
  7. 2021Graph Unlearningcited by 129
  8. 2022BackdoorBench: A Comprehensive Benchmark of Backdoor Learningcited by 124
  9. 2021SPECTRE: Defending Against Backdoor Attacks Using Robust Statisticscited by 123
  10. 2021BadEncoder: Backdoor Attacks to Pre-trained Encoders in Self-Supervised Learningcited by 120
  11. 2021Interpretable Deep Learning: Interpretation, Interpretability, Trustworthiness, and Beyondcited by 116
  12. 2023BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learningcited by 112
  13. 2024Secret Collusion among AI Agents: Multi-Agent Deception via Steganographycited by 107
  14. 2021Backdoor Attacks on Pre-trained Models by Layerwise Weight Poisoningcited by 103
  15. 2022Narcissus: A Practical Clean-Label Backdoor Attack with Limited Informationcited by 102
  16. 2022How to Backdoor Diffusion Models?cited by 102
  17. 2021Sleeper Agent: Scalable Hidden Trigger Backdoors for Neural Networks Trained from Scratchcited by 98
  18. 2021RAP: Robustness-Aware Perturbations for Defending against Backdoor Attacks on NLP Modelscited by 97
  19. 2023Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmarkcited by 97
  20. 2021Backdoor Scanning for Deep Neural Networks through K-Arm Optimizationcited by 96
  21. 2021Be Careful about Poisoned Word Embeddings: Exploring the Vulnerability of the Embedding Layers in NLP Modelscited by 96
  22. 2022Federated Unlearning: How to Efficiently Erase a Client in FL?cited by 90
  23. 2021Backdoor Attacks on Self-Supervised Learningcited by 89
  24. 2023Prompt as Triggers for Backdoor Attack: Examining the Vulnerability in Language Modelscited by 87
  25. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 86
  26. 2021Turn the Combination Lock: Learnable Textual Backdoor Attacks via Word Substitutioncited by 85
  27. 2024BadEdit: Backdooring large language models by model editingcited by 84
  28. 2024VL-Trojan: Multimodal Instruction Backdoor Attacks against Autoregressive Visual Language Modelscited by 83
  29. 2021Backdoor Attack in the Physical Worldcited by 82
  30. 2022BppAttack: Stealthy and Efficient Trojan Attacks against Deep Neural Networks via Image Quantization and Contrastive Adversarial Learningcited by 82
  31. 2021Backdoor Pre-trained Models Can Transfer to Allcited by 79
  32. 2021BadPre: Task-agnostic Backdoor Attacks to Pre-trained NLP Foundation Modelscited by 76
  33. 2023Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilitiescited by 76
  34. 2021Manipulating SGD with Data Ordering Attackscited by 73
  35. 2023Nightshade: Prompt-Specific Poisoning Attacks on Text-to-Image Generative Modelscited by 72
  36. 2023Unleashing the potential of prompt engineering for large language modelscited by 72
  37. 2021Black-box Detection of Backdoor Attacks with Limited Information and Datacited by 71
  38. 2022Federated Unlearning with Knowledge Distillationcited by 70

50 most cited of 1,052 All 1,052 (JSON)