# Backdoor Attacks on Self-Supervised Learning

arXiv: https://arxiv.org/abs/2105.10123 · HTML: https://mlsanity.com/paper/2105.10123/

## Key facts

- Cited by 90 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 40 references that are in the map.
- First posted on arXiv 21 May 2021.

## Builds on (most cited first, 29 of 29)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain](https://mlsanity.com/paper/1708.06733/) (2017)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks](https://mlsanity.com/paper/1804.00792/) (2018)
- [STRIP: A Defence Against Trojan Attacks on Deep Neural Networks](https://mlsanity.com/paper/1902.06531/) (2019)
- [Hidden Trigger Backdoor Attacks](https://mlsanity.com/paper/1910.00033/) (2019)
- [Whitening for Self-Supervised Representation Learning](https://mlsanity.com/paper/2007.06346/) (2020)
- [Scaling and Benchmarking Self-Supervised Visual Representation Learning](https://mlsanity.com/paper/1905.01235/) (2019)
- [Self-supervised Pretraining of Visual Features in the Wild](https://mlsanity.com/paper/2103.01988/) (2021)
- [Neural Trojans](https://mlsanity.com/paper/1710.00942/) (2017)
- [Universal Litmus Patterns: Revealing Backdoor Attacks in CNNs](https://mlsanity.com/paper/1906.10842/) (2019)
- [Intriguing Properties of Contrastive Losses](https://mlsanity.com/paper/2011.02803/) (2020)
- [Mean Shift for Self-Supervised Learning](https://mlsanity.com/paper/2105.07269/) (2021)
- [CompRess: Self-Supervised Learning by Compressing Representations](https://mlsanity.com/paper/2010.14713/) (2020)
- [Poisoned classifiers are not only backdoored, they are fundamentally broken](https://mlsanity.com/paper/2010.09080/) (2020)

## Built on by (most cited first, 30 of 90)

- [Backdoor Defense via Decoupling the Training Process](https://mlsanity.com/paper/2202.03423/) (2022)
- [Poisoning and Backdooring Contrastive Learning](https://mlsanity.com/paper/2106.09667/) (2021)
- [Leaving Reality to Imagination: Robust Classification via Generated Datasets](https://mlsanity.com/paper/2302.02503/) (2023)
- [CleanCLIP: Mitigating Data Poisoning Attacks in Multimodal Contrastive Learning](https://mlsanity.com/paper/2303.03323/) (2023)
- [Rickrolling the Artist: Injecting Backdoors into Text Encoders for Text-to-Image Synthesis](https://mlsanity.com/paper/2211.02408/) (2022)
- [Detecting Backdoors in Pre-trained Encoders](https://mlsanity.com/paper/2303.15180/) (2023)
- [Test-Time Backdoor Attacks on Multimodal Large Language Models](https://mlsanity.com/paper/2402.08577/) (2024)
- [Training with More Confidence: Mitigating Injected and Natural Backdoors During Training](https://mlsanity.com/paper/2202.06382/) (2022)
- [Dual-Key Multimodal Backdoors for Visual Question Answering](https://mlsanity.com/paper/2112.07668/) (2021)
- [Indiscriminate Poisoning Attacks on Unsupervised Contrastive Learning](https://mlsanity.com/paper/2202.11202/) (2022)
- [Fill-Up: Balancing Long-Tailed Data with Generative Models](https://mlsanity.com/paper/2306.07200/) (2023)
- [Downstream-agnostic Adversarial Examples](https://mlsanity.com/paper/2307.12280/) (2023)
- [An Embarrassingly Simple Backdoor Attack on Self-supervised Learning](https://mlsanity.com/paper/2210.07346/) (2022)
- [CorruptEncoder: Data Poisoning based Backdoor Attacks to Contrastive Learning](https://mlsanity.com/paper/2211.08229/) (2022)
- [ASSET: Robust Backdoor Data Detection Across a Multiplicity of Deep Learning Paradigms](https://mlsanity.com/paper/2302.11408/) (2023)
- [From Shortcuts to Triggers: Backdoor Defense with Denoised PoE](https://mlsanity.com/paper/2305.14910/) (2023)
- [Analyzing And Editing Inner Mechanisms Of Backdoored Language Models](https://mlsanity.com/paper/2302.12461/) (2023)
- [Mitigating Backdoor Poisoning Attacks through the Lens of Spurious Correlation](https://mlsanity.com/paper/2305.11596/) (2023)
- [Securely Fine-tuning Pre-trained Encoders Against Adversarial Examples](https://mlsanity.com/paper/2403.10801/) (2024)
- [Defending Against Patch-based Backdoor Attacks on Self-Supervised Learning](https://mlsanity.com/paper/2304.01482/) (2023)
- [Data and Model Poisoning Backdoor Attacks on Wireless Federated Learning, and the Defense Mechanisms: A Comprehensive Survey](https://mlsanity.com/paper/2312.08667/) (2023)
- [Backdoor Cleansing with Unlabeled Data](https://mlsanity.com/paper/2211.12044/) (2022)
- [AI Risk Management Should Incorporate Both Safety and Security](https://mlsanity.com/paper/2405.19524/) (2024)
- [Learning to Poison Large Language Models for Downstream Manipulation](https://mlsanity.com/paper/2402.13459/) (2024)
- [Stealthy Backdoor Attack in Self-Supervised Learning Vision Encoders for Large Vision Language Models](https://mlsanity.com/paper/2502.18290/) (2025)
- [Backdoor Attacks on Vision Transformers](https://mlsanity.com/paper/2206.08477/) (2022)
- [Trap and Replace: Defending Backdoor Attacks by Trapping Them into an Easy-to-Replace Subnetwork](https://mlsanity.com/paper/2210.06428/) (2022)
- [TIJO: Trigger Inversion with Joint Optimization for Defending Multimodal Backdoored Models](https://mlsanity.com/paper/2308.03906/) (2023)
- [SSL-Cleanse: Trojan Detection and Mitigation in Self-Supervised Learning](https://mlsanity.com/paper/2303.09079/) (2023)
- [Watermarking Vision-Language Pre-trained Models for Multi-modal Embedding as a Service](https://mlsanity.com/paper/2311.05863/) (2023)

## Abstract (arXiv)

Large-scale unlabeled data has spurred recent progress in self-supervised learning methods that learn rich visual representations. State-of-the-art self-supervised methods for learning representations from images (e.g., MoCo, BYOL, MSF) use an inductive bias that random augmentations (e.g., random crops) of an image should produce similar embeddings. We show that such methods are vulnerable to backdoor attacks - where an attacker poisons a small part of the unlabeled data by adding a trigger (image patch chosen by the attacker) to the images. The model performance is good on clean test images, but the attacker can manipulate the decision of the model by showing the trigger at test time. Backdoor attacks have been studied extensively in supervised learning and to the best of our knowledge, we are the first to study them for self-supervised learning. Backdoor attacks are more practical in self-supervised learning, since the use of large unlabeled data makes data inspection to remove poisons prohibitive. We show that in our targeted attack, the attacker can produce many false positives for the target category by using the trigger at test time. We also propose a defense method based on knowledge distillation that succeeds in neutralizing the attack. Our code is available here: https://github.com/UMBCvision/SSL-Backdoor .