mlsanityData to 7 Oct 2026

Detecting Backdoor Samples in Contrastive Language Image Pretraining

arXiv 2502.01385 ↗First posted 3 Feb 2025cs.LG cs.CVcited by 17

On the map: LLM safety and security → Backdoor and data poisoning attacks

Abstract (arXiv)

Contrastive language-image pretraining (CLIP) has been found to be vulnerable to poisoning backdoor attacks where the adversary can achieve an almost perfect attack success rate on CLIP models by poisoning only 0.01% of the training dataset. This raises security concerns on the current practice of pretraining large-scale models on unscrutinized web data using CLIP. In this work, we analyze the representations of backdoor-poisoned samples learned by CLIP models and find that they exhibit unique characteristics in their local subspace, i.e., their local neighborhoods are far more sparse than that of clean samples. Based on this finding, we conduct a systematic study on detecting CLIP backdoor attacks and show that these attacks can be easily and efficiently detected by traditional density ratio-based local outlier detectors, whereas existing backdoor sample detection methods fail. Our experiments also reveal that an unintentional backdoor already exists in the original CC3M dataset and has been trained into a popular open-source model released by OpenCLIP. Based on our detector, one can clean up a million-scale web dataset (e.g., CC3M) efficiently within 15 minutes using 4 Nvidia A100 GPUs. The code is publicly available in our hrefhttps://github.com/HanxunH/Detect-CLIP-Backdoor-SamplesGitHub repository.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security. First posted on arXiv 3 Feb 2025.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 10.

17 papers.

Sort:

By area:

  1. 2025X-Transfer Attacks: Towards Super Transferable Adversarial Attacks on CLIPcited by 21
  2. 2025BackdoorVLM: A Benchmark for Backdoor Attacks and Defenses on Vision-Language Modelscited by 5
  3. 2025A Dual-Purpose Framework for Backdoor Defense and Backdoor Amplification in Diffusion Modelscited by 3
  4. 2025Pre-training CLIP against Data Poisoning with Optimal Transport-based Matching and Alignmentcited by 3
  5. 2026BadCLIP++: Stealthy and Persistent Backdoors in Multimodal Contrastive Learningcited by 2
  6. 2026Dimensionality-Aware Anomaly Detection in Learned Representations of Self-Supervised Speech Modelscited by 2
  7. 2025PersGuard: Preventing Malicious Personalization in Text-to-Image Diffusion Models via Model Backdoorscited by 1
  8. 2025Geometry-Guided Adversarial Prompt Detection via Curvature and Local Intrinsic Dimensioncited by 1
  9. 2026Right Predictions, Misleading Explanations: On the Vulnerability of Vision-Language Model Explanationscited by 1
  10. 2025Assimilation Matters: Model-level Backdoor Detection in Vision-Language Pretrained Modelscited by 0
  11. 2026CLIP-Inspector: Model-Level Backdoor Detection for Prompt-Tuned CLIP via OOD Trigger Inversioncited by 0
  12. 2026TimeGuard: Channel-wise Pool Training for Backdoor Defense in Time Series Forecastingcited by 0
5 more
  1. 2026The Platonic Defense: Backdoor Defense for Self-Supervised Encoders in the Era of Large Scale Pre-trainingcited by 0
  2. 2026When Modalities Fail to Tango: Conformal Backdoor Detection in Multimodal Contrastive Learningcited by 0
  3. 2026DEFUSE: Generalizable Backdoor Defense for Self-Supervised Encoders with Generative Priorscited by 0
  4. 2026LAYERSCOPE: A Layerwise Characterization of Video and Multimodal Learned Representationscited by 0
  5. 2026TraceGuard: Adaptive Multimodal Poison Filtering through Cross-Feature Rank Agreementcited by 0