mlsanityData to 7 Oct 2026

Poisoning and Backdooring Contrastive Learning

arXiv 2106.09667 ↗First posted 17 Jun 2021cs.LGcited by 149

Not on the map. Why?

Cited by 68 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Multimodal contrastive learning methods like CLIP train on noisy and uncurated training datasets. This is cheaper than labeling datasets manually, and even improves out-of-distribution robustness. We show that this practice makes backdoor and poisoning attacks a significant threat. By poisoning just 0.01% of a dataset (e.g., just 300 images of the 3 million-example Conceptual Captions dataset), we can cause the model to misclassify test images by overlaying a small patch. Targeted poisoning attacks, whereby the model misclassifies a particular test input with an adversarially-desired label, are even easier requiring control of 0.0001% of the dataset (e.g., just three out of the 3 million images). Our attacks call into question whether training on noisy and uncurated Internet scrapes is desirable.

Key facts. Cited by 149 papers in the map (data through 7 Oct 2026). Builds on 28 of its 43 references that are in the map. First posted on arXiv 17 Jun 2021.

Builds on

28 of 43 references.

References · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2013Intriguing properties of neural networkscited by 5,114
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  8. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  9. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  10. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  11. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  12. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
16 more
  1. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  2. 2019Contrastive Multiview Codingcited by 1,297
  3. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  4. 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,052
  5. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,012
  6. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 733
  7. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  8. 2018Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networkscited by 500
  9. 2020Measuring Robustness to Natural Distribution Shifts in Image Classificationcited by 452
  10. 2019Label-Consistent Backdoor Attackscited by 285
  11. 2015Learning Visual Features from Large Weakly Supervised Datacited by 165
  12. 2021Backdoor Attacks on Self-Supervised Learningcited by 89
  13. 2021Divide and Contrast: Self-supervised Learning from Uncurated Datacited by 85
  14. 2021Poisoning the Unlabeled Dataset of Semi-Supervised Learningcited by 44
  15. 2019A Unified Framework for Data Poisoning Attack to Graph-based Semi-supervised Learningcited by 30
  16. 2018Is Data Clustering in Adversarial Settings Secure?cited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 23; 2023: 36; 2024: 40; 2025: 23; 2026: 20.

149 papers.

Sort:

By area:

  1. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 343
  2. 2021Unsolved Problems in ML Safetycited by 291
  3. 2023Poisoning Web-Scale Training Datasets is Practicalcited by 276
  4. 2023Poisoning Language Models During Instruction Tuningcited by 254
  5. 2024PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Modelscited by 251
  6. 2022Backdoor Defense via Decoupling the Training Processcited by 149
  7. 2022CyCLIP: Cyclic Contrastive Language-Image Pretrainingcited by 127
  8. 2021BadEncoder: Backdoor Attacks to Pre-trained Encoders in Self-Supervised Learningcited by 120
  9. 2023BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learningcited by 112
  10. 2023Identifying and Mitigating the Security Risks of Generative AIcited by 90
  11. 2021CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIPcited by 88
  12. 2024VL-Trojan: Multimodal Instruction Backdoor Attacks against Autoregressive Visual Language Modelscited by 83
38 more
  1. 2022BppAttack: Stealthy and Efficient Trojan Attacks against Deep Neural Networks via Image Quantization and Contrastive Adversarial Learningcited by 82
  2. 2022Untargeted Backdoor Watermark: Towards Harmless and Stealthy Dataset Copyright Protectioncited by 80
  3. 2023CleanCLIP: Mitigating Data Poisoning Attacks in Multimodal Contrastive Learningcited by 62
  4. 2022Rickrolling the Artist: Injecting Backdoors into Text Encoders for Text-to-Image Synthesiscited by 59
  5. 2023Detecting Backdoors in Pre-trained Encoderscited by 53
  6. 2024Instruction Backdoor Attacks Against Customized LLMscited by 50
  7. 2024Shadowcast: Stealthy Data Poisoning Attacks Against Vision-Language Modelscited by 48
  8. 2023Black-box Backdoor Defense via Zero-shot Image Purificationcited by 44
  9. 2021An Overview of Backdoor Attacks Against Deep Neural Networks and Possible Defencescited by 41
  10. 2022Data Poisoning Attacks Against Multimodal Encoderscited by 41
  11. 2022Training with More Confidence: Mitigating Injected and Natural Backdoors During Trainingcited by 38
  12. 2023Backdoor Defense via Deconfounded Representation Learningcited by 38
  13. 2021Dual-Key Multimodal Backdoors for Visual Question Answeringcited by 35
  14. 2024TrojVLM: Backdoor Attack Against Vision Language Modelscited by 35
  15. 2022PoisonedEncoder: Poisoning the Unlabeled Pre-training Data in Contrastive Learningcited by 33
  16. 2023NOTABLE: Transferable Backdoor Attacks Against Prompt-based NLP Modelscited by 33
  17. 2022Indiscriminate Poisoning Attacks on Unsupervised Contrastive Learningcited by 32
  18. 2023Downstream-agnostic Adversarial Examplescited by 32
  19. 2023Robust Contrastive Language-Image Pre-training against Data Poisoning and Backdoor Attackscited by 31
  20. 2024Unlearning Backdoor Threats: Enhancing Backdoor Defense in Multimodal Contrastive Learning via Local Token Unlearningcited by 31
  21. 2024Backdooring Vision-Language Models with Out-Of-Distribution Datacited by 31
  22. 2022An Embarrassingly Simple Backdoor Attack on Self-supervised Learningcited by 30
  23. 2023Towards Stable Backdoor Purification through Feature Shift Tuningcited by 30
  24. 2022Exploiting Cultural Biases via Homoglyphs in Text-to-Image Synthesiscited by 29
  25. 2023ASSET: Robust Backdoor Data Detection Across a Multiplicity of Deep Learning Paradigmscited by 29
  26. 2022CorruptEncoder: Data Poisoning based Backdoor Attacks to Contrastive Learningcited by 28
  27. 2023Nepotistically Trained Generative-AI Models Collapsecited by 27
  28. 2024Breaking the False Sense of Security in Backdoor Defense through Re-Activation Attackcited by 26
  29. 2022StolenEncoder: Stealing Pre-trained Encoders in Self-supervised Learningcited by 21
  30. 2025X-Transfer Attacks: Towards Super Transferable Adversarial Attacks on CLIPcited by 21
  31. 2023Backdoor Attacks Against Dataset Distillationcited by 20
  32. 2025Data Poisoning in Deep Learning: A Surveycited by 20
  33. 2023Defending Against Patch-based Backdoor Attacks on Self-Supervised Learningcited by 19
  34. 2023Mitigating Backdoor Poisoning Attacks through the Lens of Spurious Correlationcited by 19
  35. 2023Better Safe than Sorry: Pre-training CLIP against Targeted Data Poisoning and Backdoor Attackscited by 19
  36. 2022How to Sift Out a Clean Data Subset in the Presence of Data Poisoning?cited by 17
  37. 2024AI Risk Management Should Incorporate Both Safety and Securitycited by 17
  38. 2025Goal-oriented Backdoor Attack against Vision-Language-Action Models via Physical Objectscited by 16

50 most cited of 149 All 149 (JSON)