# CleanerCLIP: Fine-grained Counterfactual Semantic Augmentation for Backdoor Defense in Contrastive Learning

arXiv: https://arxiv.org/abs/2409.17601 · HTML: https://mlsanity.com/paper/2409.17601/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security.
- Builds on 19 of its 50 references that are in the map.
- First posted on arXiv 26 Sep 2024.

## Placed in

- LLM safety and security / [Backdoor and data poisoning attacks](https://mlsanity.com/safety/backdoor-and-data-poisoning-attacks/)

## Builds on (most cited first, 19 of 19)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain](https://mlsanity.com/paper/1708.06733/) (2017)
- [Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning](https://mlsanity.com/paper/1712.05526/) (2017)
- [Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networks](https://mlsanity.com/paper/2101.05930/) (2021)
- [Backdoor Defense via Decoupling the Training Process](https://mlsanity.com/paper/2202.03423/) (2022)
- [Poisoning and Backdooring Contrastive Learning](https://mlsanity.com/paper/2106.09667/) (2021)
- [Backdoor Attacks and Countermeasures on Deep Learning: A Comprehensive Review](https://mlsanity.com/paper/2007.10760/) (2020)
- [BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learning](https://mlsanity.com/paper/2311.12075/) (2023)
- [Data Poisoning Attacks Against Multimodal Encoders](https://mlsanity.com/paper/2209.15266/) (2022)
- [Neural Polarizer: A Lightweight and Effective Backdoor Defense via Purifying Poisoned Features](https://mlsanity.com/paper/2306.16697/) (2023)
- [On the Generalization of Multi-modal Contrastive Learning](https://mlsanity.com/paper/2306.04272/) (2023)
- [Dual Contrastive Learning: Text Classification via Label-Aware Data Augmentation](https://mlsanity.com/paper/2201.08702/) (2022)
- [Better Safe than Sorry: Pre-training CLIP against Targeted Data Poisoning and Backdoor Attacks](https://mlsanity.com/paper/2310.05862/) (2023)
- [EPSANet: An Efficient Pyramid Squeeze Attention Block on Convolutional Neural Network](https://mlsanity.com/paper/2105.14447/) (2021)
- [On Similarity](https://mlsanity.com/paper/2111.02803/) (2021)
- [WaNet -- Imperceptible Warping-based Backdoor Attack](https://mlsanity.com/paper/2102.10369/) (2021)

## Built on by (most cited first, 13 of 13)

- [Perturb and Recover: Fine-tuning for Effective Backdoor Removal from CLIP](https://mlsanity.com/paper/2412.00727/) (2024)
- [Pre-training CLIP against Data Poisoning with Optimal Transport-based Matching and Alignment](https://mlsanity.com/paper/2509.18717/) (2025)
- [Learning Noise-Resilient and Transferable Graph-Text Alignment via Dynamic Quality Assessment](https://mlsanity.com/paper/2510.19384/) (2025)
- [BadCLIP++: Stealthy and Persistent Backdoors in Multimodal Contrastive Learning](https://mlsanity.com/paper/2602.17168/) (2026)
- [InverTune: Removing Backdoors from Multimodal Contrastive Learning Models via Trigger Inversion and Activation Tuning](https://mlsanity.com/paper/2506.12411/) (2025)
- [Targeted Bit-Flip Attacks on LLM-Based Agents](https://mlsanity.com/paper/2603.10042/) (2026)
- [Assimilation Matters: Model-level Backdoor Detection in Vision-Language Pretrained Models](https://mlsanity.com/paper/2512.00343/) (2025)
- [Backdoor Attacks on Multi-modal Contrastive Learning](https://mlsanity.com/paper/2601.11006/) (2026)
- [CoDCL: Counterfactual-Inspired Augmentation Contrastive Learning for Temporal Link Prediction in Social Networks](https://mlsanity.com/paper/2601.22427/) (2026)
- [Contrastive Learning for Privacy Enhancements in Industrial Internet of Things](https://mlsanity.com/paper/2602.00515/) (2026)
- [When Modalities Fail to Tango: Conformal Backdoor Detection in Multimodal Contrastive Learning](https://mlsanity.com/paper/2608.04052/) (2026)
- [TraceGuard: Adaptive Multimodal Poison Filtering through Cross-Feature Rank Agreement](https://mlsanity.com/paper/2609.29099/) (2026)
- [Region-Level Black-Box Defense Against Stealthy Embedding-Space Backdoors in CLIP](https://mlsanity.com/paper/2609.31558/) (2026)

## Abstract (arXiv)

Pre-trained large models for multimodal contrastive learning, such as CLIP, have been widely recognized in the industry as highly susceptible to data-poisoned backdoor attacks. This poses significant risks to downstream model training. In response to such potential threats, finetuning offers a simpler and more efficient defense choice compared to retraining large models with augmented data. In the supervised learning domain, fine-tuning defense strategies can achieve excellent defense performance. However, in the unsupervised and semi-supervised domain, we find that when CLIP faces some complex attack techniques, the existing fine-tuning defense strategy, CleanCLIP, has some limitations on defense performance. The synonym substitution of its text-augmentation is insufficient to enhance the text feature space. To compensate for this weakness, we improve it by proposing a fine-grained Text Alignment Cleaner (TA-Cleaner) to cut off feature connections of backdoor triggers. We randomly select a few samples for positive and negative subtext generation at each epoch of CleanCLIP, and align the subtexts to the images to strengthen the text self-supervision. We evaluate the effectiveness of our TA-Cleaner against six attack algorithms and conduct comprehensive zero-shot classification tests on ImageNet1K. Our experimental results demonstrate that TA-Cleaner achieves state-of-the-art defensiveness among finetuning-based defense techniques. Even when faced with the novel attack technique BadCLIP, our TA-Cleaner outperforms CleanCLIP by reducing the ASR of Top-1 and Top-10 by 52.02% and 63.88%, respectively.