mlsanityData to 14 Oct 2026

Data Poisoning Attacks Against Multimodal Encoders

arXiv 2209.15266 ↗First posted 30 Sep 2022cs.CR cs.LGcited by 41

On the map: LLM safety and security → Vision-language and text-to-image safety

Abstract (arXiv)

Recently, the newly emerged multimodal models, which leverage both visual and linguistic modalities to train powerful encoders, have gained increasing attention. However, learning from a large-scale unlabeled dataset also exposes the model to the risk of potential poisoning attacks, whereby the adversary aims to perturb the model's training data to trigger malicious behaviors in it. In contrast to previous work, only poisoning visual modality, in this work, we take the first step to studying poisoning attacks against multimodal models in both visual and linguistic modalities. Specially, we focus on answering two questions: (1) Is the linguistic modality also vulnerable to poisoning attacks? and (2) Which modality is most vulnerable? To answer the two questions, we propose three types of poisoning attacks against multimodal models. Extensive evaluations on different datasets and model architectures show that all three attacks can achieve significant attack performance while maintaining model utility in both visual and linguistic modalities. Furthermore, we observe that the poisoning effect differs between different modalities. To mitigate the attacks, we propose both pre-training and post-training defenses. We empirically show that both defenses can significantly reduce the attack performance while preserving the model's utility.

Key facts. Cited by 41 papers in the map (data through 14 Oct 2026). Placed in Vision-language and text-to-image safety, one of 12 directions in LLM safety and security. Builds on 35 of its 42 references that are in the map. First posted on arXiv 30 Sep 2022.

Builds on

35 of 42 references.

References · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  9. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,849
  10. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  11. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  12. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
23 more
  1. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  2. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 991
  3. 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 953
  4. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 694
  5. 2018Improving Transferability of Adversarial Examples with Input Diversitycited by 599
  6. 2021ClipCap: CLIP Prefix for Image Captioningcited by 559
  7. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 436
  8. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 429
  9. 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 251
  10. 2018Manipulating Machine Learning: Poisoning Attacks and Countermeasures for Regression Learningcited by 225
  11. 2021CLIP2Video: Mastering Video-Text Retrieval via Image CLIPcited by 225
  12. 2020Membership Leakage in Label-Only Exposurescited by 168
  13. 2021Poisoning and Backdooring Contrastive Learningcited by 151
  14. 2019Transferable Clean-Label Poisoning Attacks on Deep Neural Netscited by 131
  15. 2022Image-text Retrieval: A Survey on Recent Research and Developmentcited by 55
  16. 2019Towards Unsupervised Image Captioning with Shared Multimodal Embeddingscited by 54
  17. 2021Quantifying and Mitigating Privacy Risks of Contrastive Learningcited by 36
  18. 2021Property Inference Attacks Against GANscited by 33
  19. 2018Data Poisoning Attacks against Online Learningcited by 27
  20. 2022Auditing Membership Leakages of Multi-Exit Networkscited by 27
  21. 2018Data Poisoning Attack against Unsupervised Node Embedding Methodscited by 19
  22. 2022Membership-Doctor: Comprehensive Assessment of Membership Inference Against Machine Learning Modelscited by 16
  23. 2018Modality Attention for End-to-End Audio-visual Speech Recognitioncited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 18; 2025: 13; 2026: 6.

41 papers.

Sort:

By area:

  1. 2023BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learningcited by 115
  2. 2023Nightshade: Prompt-Specific Poisoning Attacks on Text-to-Image Generative Modelscited by 75
  3. 2024Shadowcast: Stealthy Data Poisoning Attacks Against Vision-Language Modelscited by 50
  4. 2024Test-Time Backdoor Attacks on Multimodal Large Language Modelscited by 40
  5. 2024TrojVLM: Backdoor Attack Against Vision Language Modelscited by 39
  6. 2024Backdooring Vision-Language Models with Out-Of-Distribution Datacited by 33
  7. 2023Robust Contrastive Language-Image Pre-training against Data Poisoning and Backdoor Attackscited by 31
  8. 2024SeqMIA: Sequential-Metric Based Membership Inference Attackcited by 23
  9. 2023Better Safe than Sorry: Pre-training CLIP against Targeted Data Poisoning and Backdoor Attackscited by 19
  10. 2025Backdoor Attack on Vision Language Models with Stealthy Semantic Manipulationcited by 14
  11. 2024CleanerCLIP: Fine-grained Counterfactual Semantic Augmentation for Backdoor Defense in Contrastive Learningcited by 13
  12. 2024Defending Multimodal Backdoored Models by Repulsive Visual Prompt Tuningcited by 12
29 more
  1. 2024SOS! Soft Prompt Attack Against Open-Source Large Language Modelscited by 10
  2. 2024CL-Attack: Textual Backdoor Attacks via Cross-Lingual Triggerscited by 8
  3. 2024Semantic Shield: Defending Vision-Language Models Against Backdooring and Poisoning via Fine-grained Knowledge Alignmentcited by 7
  4. 2024Understanding Implosion in Text-to-Image Generative Modelscited by 6
  5. 2024TrojFM: Resource-efficient Backdoor Attacks against Very Large Foundation Modelscited by 5
  6. 2025Retrievals Can Be Detrimental: Unveiling the Backdoor Vulnerability of Retrieval-Augmented Diffusion Modelscited by 5
  7. 2025Evaluating and Improving Robustness in Large Language Models: A Survey and Future Directionscited by 5
  8. 2025TokenSwap: Backdoor Attack on the Compositional Understanding of Large Vision-Language Modelscited by 5
  9. 2024Perturb and Recover: Fine-tuning for Effective Backdoor Removal from CLIPcited by 4
  10. 2025ToxicTextCLIP: Text-Based Poisoning and Backdoor Attacks on CLIP Pre-trainingcited by 4
  11. 2024The Effect of Data Poisoning on Counterfactual Explanationscited by 3
  12. 2025T2UE: Generating Unlearnable Examples from Text Descriptionscited by 3
  13. 2025Pre-training CLIP against Data Poisoning with Optimal Transport-based Matching and Alignmentcited by 3
  14. 2025SWAP: Towards Copyright Auditing of Soft Prompts via Sequential Watermarkingcited by 3
  15. 2024Adversarial Attacks and Defenses in Fault Detection and Diagnosis: A Comprehensive Benchmark on the Tennessee Eastman Processcited by 2
  16. 2024New Emerged Security and Privacy of Pre-trained Model: a Survey and Outlookcited by 2
  17. 2026BadCLIP++: Stealthy and Persistent Backdoors in Multimodal Contrastive Learningcited by 2
  18. 2026Self-Purification Mitigates Backdoors in Multimodal Diffusion Language Modelscited by 2
  19. 2024Towards Sustainable SecureML: Quantifying Carbon Footprint of Adversarial Machine Learningcited by 1
  20. 2025Cyber Shadows: Neutralizing Security Threats with AI and Targeted Policy Measurescited by 1
  21. 2025Backdooring CLIP through Concept Confusioncited by 1
  22. 2025Where Do Backdoors Live? A Component-Level Analysis of Backdoor Propagation in Speech Language Modelscited by 1
  23. 2025Backdoor Unlearning by Linear Task Decompositioncited by 1
  24. 2024Unraveling Attacks in Machine Learning-based IoT Ecosystems: A Survey and the Open Libraries Behind Themcited by 0
  25. 2025Privacy-Shielded Image Compression: Defending Against Exploitation from Vision-Language Pretrained Modelscited by 0
  26. 2026Adjustable Text-Guided Backdoor Attacks with Natural-Word Triggers on Multimodal Pretrained Modelscited by 0
  27. 2026Beyond Native Success: Auditing Deployment-Interface Exposure of CLIP Backdoorscited by 0
  28. 2026When Modalities Fail to Tango: Conformal Backdoor Detection in Multimodal Contrastive Learningcited by 0
  29. 2026TraceGuard: Adaptive Multimodal Poison Filtering through Cross-Feature Rank Agreementcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Forcing Generative Models to Degenerate Ones: The Power of Data Poisoning Attacks
  2. 2025Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples
  3. 2025Beyond Text: Multimodal Jailbreaking of Vision-Language and Audio Models through Perceptually Simple Transformations
  4. 2025Investigating Vulnerabilities and Defenses Against Audio-Visual Attacks: A Comprehensive Survey Emphasizing Multimodal Models
  5. 2024Seeing is Deceiving: Exploitation of Visual Pathways in Multi-Modal Language Models
  6. 2025Con Instruction: Universal Jailbreaking of Multimodal Large Language Models via Non-Textual Modalities
  7. 2024PoisonBench: Assessing Large Language Model Vulnerability to Data Poisoning
  8. 2026Adversarial Attacks on Multimodal Large Language Models: A Comprehensive Survey