mlsanityData to 7 Oct 2026

Defending Against Patch-based Backdoor Attacks on Self-Supervised Learning

arXiv 2304.01482 ↗First posted 4 Apr 2023cs.CVcited by 19

Not on the map. Why?

Abstract (arXiv)

Recently, self-supervised learning (SSL) was shown to be vulnerable to patch-based data poisoning backdoor attacks. It was shown that an adversary can poison a small part of the unlabeled data so that when a victim trains an SSL model on it, the final model will have a backdoor that the adversary can exploit. This work aims to defend self-supervised learning against such attacks. We use a three-step defense pipeline, where we first train a model on the poisoned data. In the second step, our proposed defense algorithm (PatchSearch) uses the trained model to search the training data for poisoned samples and removes them from the training set. In the third step, a final model is trained on the cleaned-up training set. Our results show that PatchSearch is an effective defense. As an example, it improves a model's accuracy on images containing the trigger from 38.2% to 63.7% which is very close to the clean model's accuracy, 64.6%. Moreover, we show that PatchSearch outperforms baselines and state-of-the-art defense approaches including those using additional clean, trusted data. Our code is available at https://github.com/UCDvision/PatchSearch

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 50 of its 62 references that are in the map. First posted on arXiv 4 Apr 2023.

Builds on

50 of 62 references.

References · 50

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  8. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  9. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  10. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  11. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  12. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
38 more
  1. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  2. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  3. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  4. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  5. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  6. 2019Contrastive Multiview Codingcited by 1,297
  7. 2017Random Erasing Data Augmentationcited by 1,235
  8. 2016Colorful Image Colorizationcited by 1,113
  9. 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,052
  10. 2020What Makes for Good Views for Contrastive Learning?cited by 843
  11. 2018Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networkscited by 550
  12. 2017Adversarial Patchcited by 514
  13. 2018Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networkscited by 500
  14. 2018Spectral Signatures in Backdoor Attackscited by 457
  15. 2019STRIP: A Defence Against Trojan Attacks on Deep Neural Networkscited by 437
  16. 2018Detecting Backdoor Attacks on Deep Neural Networks by Activation Clusteringcited by 429
  17. 2020Backdoor Learning: A Surveycited by 412
  18. 2019Hidden Trigger Backdoor Attackscited by 359
  19. 2014Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networkscited by 299
  20. 2021Anti-Backdoor Learning: Training Clean Models on Poisoned Datacited by 258
  21. 2021Self-supervised Pretraining of Visual Features in the Wildcited by 197
  22. 2021Rethinking the Backdoor Attacks' Triggers: A Frequency Perspectivecited by 155
  23. 2017Neural Trojanscited by 151
  24. 2021Poisoning and Backdooring Contrastive Learningcited by 149
  25. 2022Backdoor Defense via Decoupling the Training Processcited by 149
  26. 2018SentiNet: Detecting Localized Universal Attacks Against Deep Learning Systemscited by 148
  27. 2019Universal Litmus Patterns: Revealing Backdoor Attacks in CNNscited by 143
  28. 2021Backdoor Attacks on Self-Supervised Learningcited by 89
  29. 2020Strong Data Augmentation Sanitizes Poisoning and Backdoor Attacks Without an Accuracy Tradeoffcited by 86
  30. 2020On the Effectiveness of Mitigating Data Poisoning Attacks with Gradient Shapingcited by 82
  31. 2021Mean Shift for Self-Supervised Learningcited by 65
  32. 2020CompRess: Self-Supervised Learning by Compressing Representationscited by 61
  33. 2019NeuronInspect: Detecting Backdoors in Neural Networks via Output Explanationscited by 49
  34. 2019Role of Spatial Context in Adversarial Robustness for Object Detectioncited by 35
  35. 2020One-pixel Signature: Characterizing CNN Models for Backdoor Detectioncited by 34
  36. 2021Topological Detection of Trojaned Neural Networkscited by 34
  37. 2020ISD: Self-Supervised Learning by Iterative Similarity Distillationcited by 33
  38. 2019Februus: Input Purification Defense Against Trojan Attacks on Deep Neural Network Systemscited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 7; 2025: 4; 2026: 3.

19 papers.

Sort:

By area:

  1. 2023BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learningcited by 112
  2. 2023A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validationcited by 91
  3. 2025Analysing the Robustness of Vision-Language-Models to Common Corruptionscited by 16
  4. 2023GhostEncoder: Stealthy Backdoor Attacks with Dynamic Triggers to Pre-trained Encoders in Self-supervised Learningcited by 6
  5. 2025Backdooring Self-Supervised Contrastive Learning by Noisy Alignmentcited by 6
  6. 2024Invisible Backdoor Attack against Self-supervised Learningcited by 5
  7. 2025Dataset Ownership Verification for Pre-trained Masked Modelscited by 2
  8. 2023Erasing Self-Supervised Learning Backdoor by Cluster Activation Maskingcited by 1
  9. 2024How to Craft Backdoors with Unlabeled Data Alone?cited by 1
  10. 2024Distributional Black-Box Model Inversion Attack with Multi-Agent Reinforcement Learningcited by 1
  11. 2024Using Interleaved Ensemble Unlearning to Keep Backdoors at Bay for Finetuning Vision Transformerscited by 1
  12. 2023Trustworthy Large Models in Vision: A Surveycited by 0
7 more
  1. 2024Towards Adversarial Robustness And Backdoor Mitigation in SSLcited by 0
  2. 2024Revealing Vulnerabilities of Neural Networks in Parameter Learning and Defense Against Explanation-Aware Backdoorscited by 0
  3. 2024Pre-trained Encoder Inference: Revealing Upstream Encoders In Downstream Machine Learning Servicescited by 0
  4. 2025Prototype Guided Backdoor Defensecited by 0
  5. 2026CLIP-Inspector: Model-Level Backdoor Detection for Prompt-Tuned CLIP via OOD Trigger Inversioncited by 0
  6. 2026Securing Self-supervised Data Curation for Foundation Models Robustnesscited by 0
  7. 2026The Platonic Defense: Backdoor Defense for Self-Supervised Encoders in the Era of Large Scale Pre-trainingcited by 0