mlsanityData to 7 Oct 2026

ToxicTextCLIP: Text-Based Poisoning and Backdoor Attacks on CLIP Pre-training

arXiv 2511.00446 ↗First posted 1 Nov 2025cs.CV cs.CR cs.LGcited by 3

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

The Contrastive Language-Image Pretraining (CLIP) model has significantly advanced vision-language modeling by aligning image-text pairs from large-scale web data through self-supervised contrastive learning. Yet, its reliance on uncurated Internet-sourced data exposes it to data poisoning and backdoor risks. While existing studies primarily investigate image-based attacks, the text modality, which is equally central to CLIP's training, remains underexplored. In this work, we introduce ToxicTextCLIP, a framework for generating high-quality adversarial texts that target CLIP during the pre-training phase. The framework addresses two key challenges: semantic misalignment caused by background inconsistency with the target class, and the scarcity of background-consistent texts. To this end, ToxicTextCLIP iteratively applies: 1) a background-aware selector that prioritizes texts with background content aligned to the target class, and 2) a background-driven augmenter that generates semantically coherent and diverse poisoned samples. Extensive experiments on classification and retrieval tasks show that ToxicTextCLIP achieves up to 95.83% poisoning success and 98.68% backdoor Hit@1, while bypassing RoCLIP, CleanCLIP and SafeCLIP defenses. The source code can be accessed via https://github.com/xinyaocse/ToxicTextCLIP/.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 35 of its 40 references that are in the map. First posted on arXiv 1 Nov 2025.

Builds on

35 of 40 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models19 Feb
  2. 2023BadCLIP: Trigger-Aware Prompt Learning for Backdoor Attacks on CLIP26 Nov
  3. 2023Robust Contrastive Language-Image Pre-training against Data Poisoning and Backdoor Attacks13 Mar

Other references · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  8. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  9. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  11. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  12. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
20 more
  1. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  2. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  3. 2020Weight Poisoning Attacks on Pre-trained Modelscited by 318
  4. 2023Poisoning Web-Scale Training Datasets is Practicalcited by 276
  5. 2020ONION: A Simple and Effective Defense Against Textual Backdoor Attackscited by 231
  6. 2021CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoderscited by 186
  7. 2021Poisoning and Backdooring Contrastive Learningcited by 149
  8. 2021BadEncoder: Backdoor Attacks to Pre-trained Encoders in Self-Supervised Learningcited by 120
  9. 2020MetaPoison: Practical General-purpose Clean-label Data Poisoningcited by 114
  10. 2023BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learningcited by 112
  11. 2023AdvCLIP: Downstream-agnostic Adversarial Examples in Multimodal Contrastive Learningcited by 101
  12. 2016Learning Visual N-Grams from Web Datacited by 95
  13. 2022MVP: Multimodality-guided Visual Pre-trainingcited by 92
  14. 2022CLIP-Art: Contrastive Pre-training for Fine-Grained Art Classificationcited by 65
  15. 2023CleanCLIP: Mitigating Data Poisoning Attacks in Multimodal Contrastive Learningcited by 62
  16. 2022Data Poisoning Attacks Against Multimodal Encoderscited by 41
  17. 2022Non-Contrastive Learning Meets Language-Image Pre-Trainingcited by 25
  18. 2024RWKV-CLIP: A Robust Vision-Language Representation Learnercited by 23
  19. 2023Better Safe than Sorry: Pre-training CLIP against Targeted Data Poisoning and Backdoor Attackscited by 19
  20. 2023Mutual-modality Adversarial Attack with Semantic Perturbationcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Adjustable Text-Guided Backdoor Attacks with Natural-Word Triggers on Multimodal Pretrained Modelscited by 0
  2. 2026Beyond Native Success: Auditing Deployment-Interface Exposure of CLIP Backdoorscited by 0
  3. 2026TraceGuard: Adaptive Multimodal Poison Filtering through Cross-Feature Rank Agreementcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Zero-Shot Defense Against Toxic Images via Inherent Multimodal Alignment in LVLMs
  2. 2025On the Feasibility of Poisoning Text-to-Image AI Models via Adversarial Mislabeling
  3. 2024Perturb and Recover: Fine-tuning for Effective Backdoor Removal from CLIP
  4. 2025LeakyCLIP: Extracting Training Data from CLIP
  5. 2024White-box Multimodal Jailbreaks Against Large Vision-Language Models
  6. 2024Revisiting the Adversarial Robustness of Vision Language Models: a Multimodal Perspective
  7. 2026Reading Between the Pixels: An Inscriptive Jailbreak Attack on Text-to-Image Models
  8. 2023Adversarial Nibbler: A Data-Centric Challenge for Improving the Safety of Text-to-Image Models