# Data Poisoning Attacks Against Multimodal Encoders

arXiv: https://arxiv.org/abs/2209.15266 · HTML: https://mlsanity.com/paper/2209.15266/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Placed in Vision-language and text-to-image safety, one of 12 directions in LLM safety and security.
- Builds on 35 of its 42 references that are in the map.
- First posted on arXiv 30 Sep 2022.

## Placed in

- LLM safety and security / [Vision-language and text-to-image safety](https://mlsanity.com/safety/vision-language-and-text-to-image-safety/)

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [Adversarial Examples Are Not Bugs, They Are Features](https://mlsanity.com/paper/1905.02175/) (2019)
- [CogView: Mastering Text-to-Image Generation via Transformers](https://mlsanity.com/paper/2105.13290/) (2021)
- [Improving Transferability of Adversarial Examples with Input Diversity](https://mlsanity.com/paper/1803.06978/) (2018)
- [ClipCap: CLIP Prefix for Image Captioning](https://mlsanity.com/paper/2111.09734/) (2021)
- [SLIP: Self-supervision meets Language-Image Pre-training](https://mlsanity.com/paper/2112.12750/) (2021)
- [VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text](https://mlsanity.com/paper/2104.11178/) (2021)
- [DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations](https://mlsanity.com/paper/2006.03659/) (2020)
- [Manipulating Machine Learning: Poisoning Attacks and Countermeasures for Regression Learning](https://mlsanity.com/paper/1804.00308/) (2018)
- [CLIP2Video: Mastering Video-Text Retrieval via Image CLIP](https://mlsanity.com/paper/2106.11097/) (2021)
- [Membership Leakage in Label-Only Exposures](https://mlsanity.com/paper/2007.15528/) (2020)
- [Poisoning and Backdooring Contrastive Learning](https://mlsanity.com/paper/2106.09667/) (2021)
- [Transferable Clean-Label Poisoning Attacks on Deep Neural Nets](https://mlsanity.com/paper/1905.05897/) (2019)
- [Image-text Retrieval: A Survey on Recent Research and Development](https://mlsanity.com/paper/2203.14713/) (2022)
- [Towards Unsupervised Image Captioning with Shared Multimodal Embeddings](https://mlsanity.com/paper/1908.09317/) (2019)
- [Quantifying and Mitigating Privacy Risks of Contrastive Learning](https://mlsanity.com/paper/2102.04140/) (2021)
- [Property Inference Attacks Against GANs](https://mlsanity.com/paper/2111.07608/) (2021)

## Built on by (most cited first, 30 of 41)

- [BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learning](https://mlsanity.com/paper/2311.12075/) (2023)
- [Nightshade: Prompt-Specific Poisoning Attacks on Text-to-Image Generative Models](https://mlsanity.com/paper/2310.13828/) (2023)
- [Shadowcast: Stealthy Data Poisoning Attacks Against Vision-Language Models](https://mlsanity.com/paper/2402.06659/) (2024)
- [Test-Time Backdoor Attacks on Multimodal Large Language Models](https://mlsanity.com/paper/2402.08577/) (2024)
- [TrojVLM: Backdoor Attack Against Vision Language Models](https://mlsanity.com/paper/2409.19232/) (2024)
- [Backdooring Vision-Language Models with Out-Of-Distribution Data](https://mlsanity.com/paper/2410.01264/) (2024)
- [Robust Contrastive Language-Image Pre-training against Data Poisoning and Backdoor Attacks](https://mlsanity.com/paper/2303.06854/) (2023)
- [SeqMIA: Sequential-Metric Based Membership Inference Attack](https://mlsanity.com/paper/2407.15098/) (2024)
- [Better Safe than Sorry: Pre-training CLIP against Targeted Data Poisoning and Backdoor Attacks](https://mlsanity.com/paper/2310.05862/) (2023)
- [Backdoor Attack on Vision Language Models with Stealthy Semantic Manipulation](https://mlsanity.com/paper/2506.07214/) (2025)
- [CleanerCLIP: Fine-grained Counterfactual Semantic Augmentation for Backdoor Defense in Contrastive Learning](https://mlsanity.com/paper/2409.17601/) (2024)
- [Defending Multimodal Backdoored Models by Repulsive Visual Prompt Tuning](https://mlsanity.com/paper/2412.20392/) (2024)
- [SOS! Soft Prompt Attack Against Open-Source Large Language Models](https://mlsanity.com/paper/2407.03160/) (2024)
- [CL-Attack: Textual Backdoor Attacks via Cross-Lingual Triggers](https://mlsanity.com/paper/2412.19037/) (2024)
- [Semantic Shield: Defending Vision-Language Models Against Backdooring and Poisoning via Fine-grained Knowledge Alignment](https://mlsanity.com/paper/2411.15673/) (2024)
- [Understanding Implosion in Text-to-Image Generative Models](https://mlsanity.com/paper/2409.12314/) (2024)
- [TrojFM: Resource-efficient Backdoor Attacks against Very Large Foundation Models](https://mlsanity.com/paper/2405.16783/) (2024)
- [Retrievals Can Be Detrimental: Unveiling the Backdoor Vulnerability of Retrieval-Augmented Diffusion Models](https://mlsanity.com/paper/2501.13340/) (2025)
- [Evaluating and Improving Robustness in Large Language Models: A Survey and Future Directions](https://mlsanity.com/paper/2506.11111/) (2025)
- [TokenSwap: Backdoor Attack on the Compositional Understanding of Large Vision-Language Models](https://mlsanity.com/paper/2509.24566/) (2025)
- [Perturb and Recover: Fine-tuning for Effective Backdoor Removal from CLIP](https://mlsanity.com/paper/2412.00727/) (2024)
- [ToxicTextCLIP: Text-Based Poisoning and Backdoor Attacks on CLIP Pre-training](https://mlsanity.com/paper/2511.00446/) (2025)
- [The Effect of Data Poisoning on Counterfactual Explanations](https://mlsanity.com/paper/2402.08290/) (2024)
- [T2UE: Generating Unlearnable Examples from Text Descriptions](https://mlsanity.com/paper/2508.03091/) (2025)
- [Pre-training CLIP against Data Poisoning with Optimal Transport-based Matching and Alignment](https://mlsanity.com/paper/2509.18717/) (2025)
- [SWAP: Towards Copyright Auditing of Soft Prompts via Sequential Watermarking](https://mlsanity.com/paper/2511.04711/) (2025)
- [Adversarial Attacks and Defenses in Fault Detection and Diagnosis: A Comprehensive Benchmark on the Tennessee Eastman Process](https://mlsanity.com/paper/2403.13502/) (2024)
- [New Emerged Security and Privacy of Pre-trained Model: a Survey and Outlook](https://mlsanity.com/paper/2411.07691/) (2024)
- [BadCLIP++: Stealthy and Persistent Backdoors in Multimodal Contrastive Learning](https://mlsanity.com/paper/2602.17168/) (2026)
- [Self-Purification Mitigates Backdoors in Multimodal Diffusion Language Models](https://mlsanity.com/paper/2602.22246/) (2026)

## Abstract (arXiv)

Recently, the newly emerged multimodal models, which leverage both visual and linguistic modalities to train powerful encoders, have gained increasing attention. However, learning from a large-scale unlabeled dataset also exposes the model to the risk of potential poisoning attacks, whereby the adversary aims to perturb the model's training data to trigger malicious behaviors in it. In contrast to previous work, only poisoning visual modality, in this work, we take the first step to studying poisoning attacks against multimodal models in both visual and linguistic modalities. Specially, we focus on answering two questions: (1) Is the linguistic modality also vulnerable to poisoning attacks? and (2) Which modality is most vulnerable? To answer the two questions, we propose three types of poisoning attacks against multimodal models. Extensive evaluations on different datasets and model architectures show that all three attacks can achieve significant attack performance while maintaining model utility in both visual and linguistic modalities. Furthermore, we observe that the poisoning effect differs between different modalities. To mitigate the attacks, we propose both pre-training and post-training defenses. We empirically show that both defenses can significantly reduce the attack performance while preserving the model's utility.