mlsanityData to 14 Oct 2026

Robust Contrastive Language-Image Pre-training against Data Poisoning and Backdoor Attacks

arXiv 2303.06854 ↗First posted 13 Mar 2023cs.CV cs.CL cs.CRcited by 31

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

Contrastive vision-language representation learning has achieved state-of-the-art performance for zero-shot classification, by learning from millions of image-caption pairs crawled from the internet. However, the massive data that powers large multimodal models such as CLIP, makes them extremely vulnerable to various types of targeted data poisoning and backdoor attacks. Despite this vulnerability, robust contrastive vision-language pre-training against such attacks has remained unaddressed. In this work, we propose ROCLIP, the first effective method for robust pre-training multimodal vision-language models against targeted data poisoning and backdoor attacks. ROCLIP effectively breaks the association between poisoned image-caption pairs by considering a relatively large and varying pool of random captions, and matching every image with the text that is most similar to it in the pool instead of its own caption, every few epochs.It also leverages image and text augmentations to further strengthen the defense and improve the performance of the model. Our extensive experiments show that ROCLIP renders state-of-the-art targeted data poisoning and backdoor attacks ineffective during pre-training CLIP models. In particular, ROCLIP decreases the success rate for targeted data poisoning attacks from 93.75% to 12.5% and that of backdoor attacks down to 0%, while improving the model's linear probe performance by 10% and maintains a similar zero shot performance compared to CLIP. By increasing the frequency of matching, ROCLIP is able to defend strong attacks, which add up to 1% poisoned examples to the data, and successfully maintain a low attack success rate of 12.5%, while trading off the performance on some tasks.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 23 of its 27 references that are in the map. First posted on arXiv 13 Mar 2023.

Builds on

23 of 27 references.

References · 23

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  6. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  7. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  8. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  9. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  10. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 807
  11. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 627
  12. 2018Do Better ImageNet Models Transfer Better?cited by 565
11 more
  1. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 436
  2. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  3. 2019Hidden Trigger Backdoor Attackscited by 371
  4. 2021With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representationscited by 326
  5. 2020Witches' Brew: Industrial Scale Data Poisoning via Gradient Matchingcited by 163
  6. 2021Poisoning and Backdooring Contrastive Learningcited by 151
  7. 2022CyCLIP: Cyclic Contrastive Language-Image Pretrainingcited by 127
  8. 2020Strong Data Augmentation Sanitizes Poisoning and Backdoor Attacks Without an Accuracy Tradeoffcited by 89
  9. 2023CleanCLIP: Mitigating Data Poisoning Attacks in Multimodal Contrastive Learningcited by 63
  10. 2021Revisiting Contrastive Methods for Unsupervised Learning of Visual Representationscited by 48
  11. 2022Data Poisoning Attacks Against Multimodal Encoderscited by 41

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 13; 2025: 7; 2026: 10.

31 papers.

Sort:

By area:

  1. 2024A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trendscited by 84
  2. 2024Breaking the False Sense of Security in Backdoor Defense through Re-Activation Attackcited by 28
  3. 2024Unbridled Icarus: A Survey of the Potential Perils of Image Inputs in Multimodal Large Language Model Securitycited by 19
  4. 2024Espresso: Robust Concept Filtering in Text-to-Image Modelscited by 14
  5. 2024Defending Multimodal Backdoored Models by Repulsive Visual Prompt Tuningcited by 12
  6. 2024Adversarial Backdoor Defense in CLIPcited by 10
  7. 2024Semantic Shield: Defending Vision-Language Models Against Backdooring and Poisoning via Fine-grained Knowledge Alignmentcited by 7
  8. 2024Data-centric NLP Backdoor Defense from the Lens of Memorizationcited by 6
  9. 2024Efficient Backdoor Defense in Multimodal Contrastive Learning: A Token-Level Unlearning Method for Mitigating Threatscited by 6
  10. 2024DeDe: Detecting Backdoor Samples for SSL Encoders via Decoderscited by 6
  11. 2024Test-Time Multimodal Backdoor Detection by Contrastive Promptingcited by 5
  12. 2025TokenSwap: Backdoor Attack on the Compositional Understanding of Large Vision-Language Modelscited by 5
19 more
  1. 2024Perturb and Recover: Fine-tuning for Effective Backdoor Removal from CLIPcited by 4
  2. 2025ToxicTextCLIP: Text-Based Poisoning and Backdoor Attacks on CLIP Pre-trainingcited by 4
  3. 2025T2UE: Generating Unlearnable Examples from Text Descriptionscited by 3
  4. 2023Effective Backdoor Mitigation in Vision-Language Models Depends on the Pre-training Objectivecited by 2
  5. 2024New Emerged Security and Privacy of Pre-trained Model: a Survey and Outlookcited by 2
  6. 2026BadCLIP++: Stealthy and Persistent Backdoors in Multimodal Contrastive Learningcited by 2
  7. 2025Neural Antidote: Class-Wise Prompt Tuning for Purifying Backdoors in CLIPcited by 1
  8. 2025InverTune: Removing Backdoors from Multimodal Contrastive Learning Models via Trigger Inversion and Activation Tuningcited by 1
  9. 2025Backdoor Unlearning by Linear Task Decompositioncited by 1
  10. 2025Assimilation Matters: Model-level Backdoor Detection in Vision-Language Pretrained Modelscited by 0
  11. 2026Hidden Ads: Behavior Triggered Semantic Backdoors for Advertisement Injection in Vision Language Modelscited by 0
  12. 2026BYORn: Bootstrap Your Own Responses to Defend Large Vision-Language Models Against Backdoor Attackscited by 0
  13. 2026Beyond Native Success: Auditing Deployment-Interface Exposure of CLIP Backdoorscited by 0
  14. 2026The Platonic Defense: Backdoor Defense for Self-Supervised Encoders in the Era of Large Scale Pre-trainingcited by 0
  15. 2026When Modalities Fail to Tango: Conformal Backdoor Detection in Multimodal Contrastive Learningcited by 0
  16. 2026DSPrompt: Dynamic Soft Prompt Defense Against M-RAG Corruptioncited by 0
  17. 2026ViTeGate: Visual-Textual Triggered Knowledge Poisoning for Vision-Language Retrieval-Augmented Generationcited by 0
  18. 2026TraceGuard: Adaptive Multimodal Poison Filtering through Cross-Feature Rank Agreementcited by 0
  19. 2026Region-Level Black-Box Defense Against Stealthy Embedding-Space Backdoors in CLIPcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Better Safe than Sorry: Pre-training CLIP against Targeted Data Poisoning and Backdoor Attacks
  2. 2024Revisiting the Adversarial Robustness of Vision Language Models: a Multimodal Perspective
  3. 2025Robust Defense Strategies for Multimodal Contrastive Learning: Efficient Fine-tuning Against Backdoor Attacks
  4. 2025Pre-training CLIP against Data Poisoning with Optimal Transport-based Matching and Alignment
  5. 2023BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learning
  6. 2026Towards Robustness against Typographic Attack with Training-free Concept Localization
  7. 2025Detecting Backdoor Samples in Contrastive Language Image Pretraining
  8. 2025Zero-Shot Defense Against Toxic Images via Inherent Multimodal Alignment in LVLMs