# Detecting Backdoor Samples in Contrastive Language Image Pretraining

arXiv: https://arxiv.org/abs/2502.01385 · HTML: https://mlsanity.com/paper/2502.01385/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security.
- First posted on arXiv 3 Feb 2025.

## Placed in

- LLM safety and security / [Backdoor and data poisoning attacks](https://mlsanity.com/safety/backdoor-and-data-poisoning-attacks/)

## Built on by (most cited first, 19 of 19)

- [X-Transfer Attacks: Towards Super Transferable Adversarial Attacks on CLIP](https://mlsanity.com/paper/2505.05528/) (2025)
- [BackdoorVLM: A Benchmark for Backdoor Attacks and Defenses on Vision-Language Models](https://mlsanity.com/paper/2511.18921/) (2025)
- [A Dual-Purpose Framework for Backdoor Defense and Backdoor Amplification in Diffusion Models](https://mlsanity.com/paper/2502.19047/) (2025)
- [Pre-training CLIP against Data Poisoning with Optimal Transport-based Matching and Alignment](https://mlsanity.com/paper/2509.18717/) (2025)
- [BadCLIP++: Stealthy and Persistent Backdoors in Multimodal Contrastive Learning](https://mlsanity.com/paper/2602.17168/) (2026)
- [Dimensionality-Aware Anomaly Detection in Learned Representations of Self-Supervised Speech Models](https://mlsanity.com/paper/2605.02715/) (2026)
- [PersGuard: Preventing Malicious Personalization in Text-to-Image Diffusion Models via Model Backdoors](https://mlsanity.com/paper/2502.16167/) (2025)
- [Geometry-Guided Adversarial Prompt Detection via Curvature and Local Intrinsic Dimension](https://mlsanity.com/paper/2503.03502/) (2025)
- [Right Predictions, Misleading Explanations: On the Vulnerability of Vision-Language Model Explanations](https://mlsanity.com/paper/2605.16651/) (2026)
- [Assimilation Matters: Model-level Backdoor Detection in Vision-Language Pretrained Models](https://mlsanity.com/paper/2512.00343/) (2025)
- [CLIP-Inspector: Model-Level Backdoor Detection for Prompt-Tuned CLIP via OOD Trigger Inversion](https://mlsanity.com/paper/2604.09101/) (2026)
- [TimeGuard: Channel-wise Pool Training for Backdoor Defense in Time Series Forecasting](https://mlsanity.com/paper/2605.22365/) (2026)
- [The Platonic Defense: Backdoor Defense for Self-Supervised Encoders in the Era of Large Scale Pre-training](https://mlsanity.com/paper/2606.29451/) (2026)
- [When Modalities Fail to Tango: Conformal Backdoor Detection in Multimodal Contrastive Learning](https://mlsanity.com/paper/2608.04052/) (2026)
- [DEFUSE: Generalizable Backdoor Defense for Self-Supervised Encoders with Generative Priors](https://mlsanity.com/paper/2608.25851/) (2026)
- [LAYERSCOPE: A Layerwise Characterization of Video and Multimodal Learned Representations](https://mlsanity.com/paper/2609.28086/) (2026)
- [TraceGuard: Adaptive Multimodal Poison Filtering through Cross-Feature Rank Agreement](https://mlsanity.com/paper/2609.29099/) (2026)
- [Space Over Statistic, Retraining Over Ranking: Auditing Training Labels Under Label-Only Backdoors](https://mlsanity.com/paper/2610.13051/) (2026)
- [DPOison-VL: Clean Preference Poisoning in Vision-Language Direct Preference Optimization](https://mlsanity.com/paper/2610.18609/) (2026)

## Abstract (arXiv)

Contrastive language-image pretraining (CLIP) has been found to be vulnerable to poisoning backdoor attacks where the adversary can achieve an almost perfect attack success rate on CLIP models by poisoning only 0.01% of the training dataset. This raises security concerns on the current practice of pretraining large-scale models on unscrutinized web data using CLIP. In this work, we analyze the representations of backdoor-poisoned samples learned by CLIP models and find that they exhibit unique characteristics in their local subspace, i.e., their local neighborhoods are far more sparse than that of clean samples. Based on this finding, we conduct a systematic study on detecting CLIP backdoor attacks and show that these attacks can be easily and efficiently detected by traditional density ratio-based local outlier detectors, whereas existing backdoor sample detection methods fail. Our experiments also reveal that an unintentional backdoor already exists in the original CC3M dataset and has been trained into a popular open-source model released by OpenCLIP. Based on our detector, one can clean up a million-scale web dataset (e.g., CC3M) efficiently within 15 minutes using 4 Nvidia A100 GPUs. The code is publicly available in our hrefhttps://github.com/HanxunH/Detect-CLIP-Backdoor-SamplesGitHub repository.