mlsanityData to 7 Oct 2026

CleanCLIP: Mitigating Data Poisoning Attacks in Multimodal Contrastive Learning

arXiv 2303.03323 ↗First posted 6 Mar 2023cs.CV cs.AI cs.CRcited by 62

On the map: LLM safety and security → Backdoor and data poisoning attacks

Abstract (arXiv)

Multimodal contrastive pretraining has been used to train multimodal representation models, such as CLIP, on large amounts of paired image-text data. However, previous studies have revealed that such models are vulnerable to backdoor attacks. Specifically, when trained on backdoored examples, CLIP learns spurious correlations between the embedded backdoor trigger and the target label, aligning their representations in the joint embedding space. Injecting even a small number of poisoned examples, such as 75 examples in 3 million pretraining data, can significantly manipulate the model's behavior, making it difficult to detect or unlearn such correlations. To address this issue, we propose CleanCLIP, a finetuning framework that weakens the learned spurious associations introduced by backdoor attacks by independently re-aligning the representations for individual modalities. We demonstrate that unsupervised finetuning using a combination of multimodal contrastive and unimodal self-supervised objectives for individual modalities can significantly reduce the impact of the backdoor attack. Additionally, we show that supervised finetuning on task-specific labeled image data removes the backdoor trigger from the CLIP vision encoder. We show empirically that CleanCLIP maintains model performance on benign examples while erasing a range of backdoor attacks on multimodal contrastive learning. The code and checkpoints are available at https://github.com/nishadsinghi/CleanCLIP.

Key facts. Cited by 62 papers in the map (data through 7 Oct 2026). Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security. Builds on 47 of its 66 references that are in the map. First posted on arXiv 6 Mar 2023.

Builds on

47 of 66 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Not All Poisons are Created Equal: Robust Training against Data Poisoning18 Oct
  2. 2022Friendly Noise against Adversarial Noise: A Powerful Defense against Data Poisoning Attacks14 Aug
  3. 2021BadEncoder: Backdoor Attacks to Pre-trained Encoders in Self-Supervised Learning1 Aug
  4. 2021Rethinking the Backdoor Attacks' Triggers: A Frequency Perspective7 Apr

Other references · 43

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  10. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  11. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  12. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
31 more
  1. 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,052
  2. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,012
  3. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  4. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 800
  5. 2018Spectral Signatures in Backdoor Attackscited by 457
  6. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  7. 2018Detecting Backdoor Attacks on Deep Neural Networks by Activation Clusteringcited by 429
  8. 2020Backdoor Learning: A Surveycited by 412
  9. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  10. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  11. 2021Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networkscited by 313
  12. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  13. 2020Input-Aware Dynamic Backdoor Attackcited by 304
  14. 2019Label-Consistent Backdoor Attackscited by 285
  15. 2023Poisoning Web-Scale Training Datasets is Practicalcited by 276
  16. 2021Anti-Backdoor Learning: Training Clean Models on Poisoned Datacited by 258
  17. 2019A new Backdoor Attack in CNNs by training set corruption without label poisoningcited by 237
  18. 2021Adversarial Neuron Pruning Purifies Backdoored Deep Modelscited by 230
  19. 2019A backdoor attack against LSTM-based text classification systemscited by 214
  20. 2020BadNL: Backdoor Attacks against NLP Models with Semantic-preserving Improvementscited by 191
  21. 2021Adversarial Unlearning of Backdoors via Implicit Hypergradientcited by 160
  22. 2021Poisoning and Backdooring Contrastive Learningcited by 149
  23. 2022Backdoor Defense via Decoupling the Training Processcited by 149
  24. 2022CyCLIP: Cyclic Contrastive Language-Image Pretrainingcited by 127
  25. 2022BackdoorBench: A Comprehensive Benchmark of Backdoor Learningcited by 124
  26. 2019Robust Anomaly Detection and Backdoor Attack Detection Via Differential Privacycited by 93
  27. 2021Backdoor Attacks on Self-Supervised Learningcited by 89
  28. 2020Strong Data Augmentation Sanitizes Poisoning and Backdoor Attacks Without an Accuracy Tradeoffcited by 86
  29. 2021Black-box Detection of Backdoor Attacks with Limited Information and Datacited by 71
  30. 2022Fine-Tuning Is All You Need to Mitigate Backdoor Attackscited by 35
  31. 2021A Fistful of Words: Learning Transferable Visual Models from Bag-of-Words Supervisioncited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 12; 2024: 19; 2025: 20; 2026: 11.

62 papers.

Sort:

By area:

  1. 2023BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learningcited by 112
  2. 2024VL-Trojan: Multimodal Instruction Backdoor Attacks against Autoregressive Visual Language Modelscited by 83
  3. 2024A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trendscited by 82
  4. 2023BadCLIP: Trigger-Aware Prompt Learning for Backdoor Attacks on CLIPcited by 63
  5. 2023Exploring the Landscape of Machine Unlearning: A Comprehensive Survey and Taxonomycited by 44
  6. 2024Test-Time Backdoor Attacks on Multimodal Large Language Modelscited by 35
  7. 2024Threats, Attacks, and Defenses in Machine Unlearning: A Surveycited by 34
  8. 2023Robust Contrastive Language-Image Pre-training against Data Poisoning and Backdoor Attackscited by 31
  9. 2024Unlearning Backdoor Threats: Enhancing Backdoor Defense in Multimodal Contrastive Learning via Local Token Unlearningcited by 31
  10. 2024Breaking the False Sense of Security in Backdoor Defense through Re-Activation Attackcited by 26
  11. 2025MM-PoisonRAG: Disrupting Multimodal RAG with Local and Global Poisoning Attackscited by 24
  12. 2023Better Safe than Sorry: Pre-training CLIP against Targeted Data Poisoning and Backdoor Attackscited by 19
38 more
  1. 2023Defenses in Adversarial Machine Learning: A Surveycited by 14
  2. 2023A Systematic Survey of Prompt Engineering on Vision-Language Foundation Modelscited by 12
  3. 2023FRAMU: Attention-based Machine Unlearning using Federated Reinforcement Learningcited by 12
  4. 2024Sim-CLIP: Unsupervised Siamese Adversarial Fine-Tuning for Robust and Semantically-Rich Vision-Language Modelscited by 12
  5. 2024Defending Multimodal Backdoored Models by Repulsive Visual Prompt Tuningcited by 12
  6. 2024Adversarial Backdoor Defense in CLIPcited by 10
  7. 2024Semantic Shield: Defending Vision-Language Models Against Backdooring and Poisoning via Fine-grained Knowledge Alignmentcited by 7
  8. 2024Efficient Backdoor Defense in Multimodal Contrastive Learning: A Token-Level Unlearning Method for Mitigating Threatscited by 6
  9. 2024DeDe: Detecting Backdoor Samples for SSL Encoders via Decoderscited by 6
  10. 2025OCRT: Boosting Foundation Models in the Open World with Object-Concept-Relation Triadcited by 6
  11. 2025Backdooring Self-Supervised Contrastive Learning by Noisy Alignmentcited by 6
  12. 2026Multimodal Unlearning Across Vision, Language, Video, and Audio: Survey of Methods, Datasets, and Benchmarkscited by 6
  13. 2024Test-Time Multimodal Backdoor Detection by Contrastive Promptingcited by 5
  14. 2025Retrievals Can Be Detrimental: Unveiling the Backdoor Vulnerability of Retrieval-Augmented Diffusion Modelscited by 5
  15. 2024TrojFM: Resource-efficient Backdoor Attacks against Very Large Foundation Modelscited by 4
  16. 2024Towards Robust Object Detection: Identifying and Removing Backdoors via Module Inconsistency Analysiscited by 4
  17. 2024Perturb and Recover: Fine-tuning for Effective Backdoor Removal from CLIPcited by 4
  18. 2025TokenSwap: Backdoor Attack on the Compositional Understanding of Large Vision-Language Modelscited by 4
  19. 2025BADTV: Unveiling Backdoor Threats in Third-Party Task Vectorscited by 3
  20. 2025Pre-training CLIP against Data Poisoning with Optimal Transport-based Matching and Alignmentcited by 3
  21. 2025Learning Noise-Resilient and Transferable Graph-Text Alignment via Dynamic Quality Assessmentcited by 3
  22. 2025ToxicTextCLIP: Text-Based Poisoning and Backdoor Attacks on CLIP Pre-trainingcited by 3
  23. 2023Effective Backdoor Mitigation in Vision-Language Models Depends on the Pre-training Objectivecited by 2
  24. 2024New Emerged Security and Privacy of Pre-trained Model: a Survey and Outlookcited by 2
  25. 2026BadCLIP++: Stealthy and Persistent Backdoors in Multimodal Contrastive Learningcited by 2
  26. 2023Unlearnable Examples Give a False Sense of Data Privacy: Understanding and Relearningcited by 1
  27. 2023Erasing Self-Supervised Learning Backdoor by Cluster Activation Maskingcited by 1
  28. 2024Transparency Attacks: How Imperceptible Image Layers Can Fool AI Perceptioncited by 1
  29. 2025Decoding Visual Neural Representations by Multimodal with Dynamic Balancingcited by 1
  30. 2025SABER: Uncovering Vulnerabilities in Safety Alignment via Cross-Layer Residual Connectioncited by 1
  31. 2025Backdoor Unlearning by Linear Task Decompositioncited by 1
  32. 2025SegDebias: Test-Time Bias Mitigation for ViT-Based CLIP via Segmentationcited by 1
  33. 2025Stealth Fine-Tuning: Efficiently Breaking Alignment in RVLMs Using Self-Generated CoTcited by 1
  34. 2026BackdoorIDS: Zero-shot Backdoor Detection for Pretrained Vision Encodercited by 1
  35. 2023Trustworthy Large Models in Vision: A Surveycited by 0
  36. 2024Exploiting Alpha Transparency In Language And Vision-Based AI Systemscited by 0
  37. 2025Neural Antidote: Class-Wise Prompt Tuning for Purifying Backdoors in CLIPcited by 0
  38. 2025BadPromptFL: A Novel Backdoor Threat to Prompt-based Federated Learning in Multimodal Modelscited by 0

50 most cited of 62 All 62 (JSON)