# CLIP-Guided Backdoor Defense through Entropy-Based Poisoned Dataset Separation

arXiv: https://arxiv.org/abs/2507.05113 · HTML: https://mlsanity.com/paper/2507.05113/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security.
- Builds on 27 of its 62 references that are in the map.
- First posted on arXiv 7 Jul 2025.

## Placed in

- LLM safety and security / [Backdoor and data poisoning attacks](https://mlsanity.com/safety/backdoor-and-data-poisoning-attacks/)

## Builds on (most cited first, 27 of 27)

- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning](https://mlsanity.com/paper/1712.05526/) (2017)
- [Invisible Backdoor Attack with Sample-Specific Triggers](https://mlsanity.com/paper/2012.03816/) (2020)
- [Input-Aware Dynamic Backdoor Attack](https://mlsanity.com/paper/2010.08138/) (2020)
- [Label-Consistent Backdoor Attacks](https://mlsanity.com/paper/1912.02771/) (2019)
- [Anti-Backdoor Learning: Training Clean Models on Poisoned Data](https://mlsanity.com/paper/2110.11571/) (2021)
- [Adversarial Neuron Pruning Purifies Backdoored Deep Models](https://mlsanity.com/paper/2110.14430/) (2021)
- [Adversarial Unlearning of Backdoors via Implicit Hypergradient](https://mlsanity.com/paper/2110.03735/) (2021)
- [BioCLIP: A Vision Foundation Model for the Tree of Life](https://mlsanity.com/paper/2311.18803/) (2023)
- [Backdoor Defense via Decoupling the Training Process](https://mlsanity.com/paper/2202.03423/) (2022)
- [Bridging Mode Connectivity in Loss Landscapes and Adversarial Robustness](https://mlsanity.com/paper/2005.00060/) (2020)
- [BackdoorBench: A Comprehensive Benchmark of Backdoor Learning](https://mlsanity.com/paper/2206.12654/) (2022)
- [CyCLIP: Cyclic Contrastive Language-Image Pretraining](https://mlsanity.com/paper/2205.14459/) (2022)
- [BppAttack: Stealthy and Efficient Trojan Attacks against Deep Neural Networks via Image Quantization and Contrastive Adversarial Learning](https://mlsanity.com/paper/2205.13383/) (2022)
- [UniCLIP: Unified Framework for Contrastive Language-Image Pre-training](https://mlsanity.com/paper/2209.13430/) (2022)
- [Backdoor Defense via Adaptively Splitting Poisoned Dataset](https://mlsanity.com/paper/2303.12993/) (2023)
- [Physical Backdoor Attacks to Lane Detection Systems in Autonomous Driving](https://mlsanity.com/paper/2203.00858/) (2022)
- [Better Safe than Sorry: Pre-training CLIP against Targeted Data Poisoning and Backdoor Attacks](https://mlsanity.com/paper/2310.05862/) (2023)
- [BackdoorBench: A Comprehensive Benchmark and Analysis of Backdoor Learning](https://mlsanity.com/paper/2407.19845/) (2024)
- [One Surrogate to Fool Them All: Universal, Transferable, and Targeted Adversarial Attacks with CLIP](https://mlsanity.com/paper/2505.19840/) (2025)
- [Backdoor Secrets Unveiled: Identifying Backdoor Data with Optimized Scaled Prediction Consistency](https://mlsanity.com/paper/2403.10717/) (2024)
- [Progressive Poisoned Data Isolation for Training-time Backdoor Defense](https://mlsanity.com/paper/2312.12724/) (2023)
- [GeomCLIP: Contrastive Geometry-Text Pre-training for Molecules](https://mlsanity.com/paper/2411.10821/) (2024)

## Built on by (most cited first, 10 of 10)

- [MedSynapse-V: Bridging Visual Perception and Clinical Intuition via Latent Memory Evolution](https://mlsanity.com/paper/2604.26283/) (2026)
- [From Multi-Agent to Single-Agent: When Is Skill Distillation Beneficial?](https://mlsanity.com/paper/2604.01608/) (2026)
- [Contextual Agentic Memory is a Memo, Not True Memory](https://mlsanity.com/paper/2604.27707/) (2026)
- [Position: Universal Time Series Foundation Models Rest on a Category Error](https://mlsanity.com/paper/2602.05287/) (2026)
- [From Internal Diagnosis to External Auditing: A VLM-Driven Paradigm for Data-Free Online Backdoor Defense](https://mlsanity.com/paper/2601.19448/) (2026)
- [When Agent Automation Becomes Profitable: Quantifying and Insuring Autonomous AI Risk through Trace-Economic Underwriting](https://mlsanity.com/paper/2606.16465/) (2026)
- [LLM Agents Are Latent Context Managers: Eliciting Self-Managed Context via State Proprioception](https://mlsanity.com/paper/2606.30005/) (2026)
- [Reviewer Scores Are Not Comparable Across Research Areas in ML Peer Review](https://mlsanity.com/paper/2607.27209/) (2026)
- [TraceGuard: Adaptive Multimodal Poison Filtering through Cross-Feature Rank Agreement](https://mlsanity.com/paper/2609.29099/) (2026)
- [TARE: Weigh a Never-Poisoned Twin Before Reading Backdoor-Defense Costs](https://mlsanity.com/paper/2610.06994/) (2026)

## Abstract (arXiv)

Deep Neural Networks (DNNs) are susceptible to backdoor attacks, where adversaries poison training data to implant backdoor into the victim model. Current backdoor defenses on poisoned data often suffer from high computational costs or low effectiveness against advanced attacks like clean-label and clean-image backdoors. To address them, we introduce CLIP-Guided backdoor Defense (CGD), an efficient and effective method that mitigates various backdoor attacks. CGD utilizes a publicly accessible CLIP model to identify inputs that are likely to be clean or poisoned. It then retrains the model with these inputs, using CLIP's logits as a guidance to effectively neutralize the backdoor. Experiments on 4 datasets and 11 attack types demonstrate that CGD reduces attack success rates (ASRs) to below 1% while maintaining clean accuracy (CA) with a maximum drop of only 0.3%, outperforming existing defenses. Additionally, we show that clean-data-based defenses can be adapted to poisoned data using CGD. Also, CGD exhibits strong robustness, maintaining low ASRs even when employing a weaker CLIP model or when CLIP itself is compromised by a backdoor. These findings underscore CGD's exceptional efficiency, effectiveness, and applicability for real-world backdoor defense scenarios. Code: https://github.com/binyxu/CGD.