mlsanityData to 7 Oct 2026

Adversarial Backdoor Defense in CLIP

arXiv 2409.15968 ↗First posted 24 Sep 2024cs.CVcited by 10

On the map: LLM safety and security → Backdoor and data poisoning attacks

Abstract (arXiv)

Multimodal contrastive pretraining, exemplified by models like CLIP, has been found to be vulnerable to backdoor attacks. While current backdoor defense methods primarily employ conventional data augmentation to create augmented samples aimed at feature alignment, these methods fail to capture the distinct features of backdoor samples, resulting in suboptimal defense performance. Observations reveal that adversarial examples and backdoor samples exhibit similarities in the feature space within the compromised models. Building on this insight, we propose Adversarial Backdoor Defense (ABD), a novel data augmentation strategy that aligns features with meticulously crafted adversarial examples. This approach effectively disrupts the backdoor association. Our experiments demonstrate that ABD provides robust defense against both traditional uni-modal and multimodal backdoor attacks targeting CLIP. Compared to the current state-of-the-art defense method, CleanCLIP, ABD reduces the attack success rate by 8.66% for BadNet, 10.52% for Blended, and 53.64% for BadCLIP, while maintaining a minimal average decrease of just 1.73% in clean accuracy.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security. Builds on 34 of its 42 references that are in the map. First posted on arXiv 24 Sep 2024.

Builds on

34 of 42 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023BadCLIP: Trigger-Aware Prompt Learning for Backdoor Attacks on CLIP26 Nov
  2. 2023BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learning20 Nov
  3. 2023TIJO: Trigger Inversion with Joint Optimization for Defending Multimodal Backdoored Models7 Aug
  4. 2023Detecting Backdoors in Pre-trained Encoders23 Mar
  5. 2023Robust Contrastive Language-Image Pre-training against Data Poisoning and Backdoor Attacks13 Mar
  6. 2023CleanCLIP: Mitigating Data Poisoning Attacks in Multimodal Contrastive Learning6 Mar
  7. 2021BadEncoder: Backdoor Attacks to Pre-trained Encoders in Self-Supervised Learning1 Aug

Other references · 27

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  4. 2022Generative Adversarial Networkscited by 5,603
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  6. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  7. 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,052
  8. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,012
  9. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 800
  10. 2021Transformer in Transformercited by 632
  11. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  12. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
15 more
  1. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  2. 2021Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networkscited by 313
  3. 2023Poisoning Web-Scale Training Datasets is Practicalcited by 276
  4. 2021Anti-Backdoor Learning: Training Clean Models on Poisoned Datacited by 258
  5. 2021Adversarial Neuron Pruning Purifies Backdoored Deep Modelscited by 230
  6. 2021Adversarial Unlearning of Backdoors via Implicit Hypergradientcited by 160
  7. 2021Poisoning and Backdooring Contrastive Learningcited by 149
  8. 2022CyCLIP: Cyclic Contrastive Language-Image Pretrainingcited by 127
  9. 2023AdvCLIP: Downstream-agnostic Adversarial Examples in Multimodal Contrastive Learningcited by 101
  10. 2023Shared Adversarial Unlearning: Backdoor Mitigation by Unlearning Shared Adversarial Examplescited by 40
  11. 2024Universal Adversarial Perturbations for Vision-Language Pre-trained Modelscited by 31
  12. 2022On the Effectiveness of Adversarial Training against Backdoor Attackscited by 15
  13. 2022Progressive Backdoor Erasing via connecting Backdoor and Adversarial Attackscited by 10
  14. 2024Towards Unified Robustness Against Both Backdoor and Adversarial Attackscited by 9
  15. 2021Detecting Backdoor in Deep Neural Networks via Intentional Adversarial Perturbationscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 5; 2026: 3.

10 papers.

  1. 2024Defending Multimodal Backdoored Models by Repulsive Visual Prompt Tuningcited by 12
  2. 2025SRD: Reinforcement-Learned Semantic Perturbation for Backdoor Defense in VLMscited by 9
  3. 2025ICLShield: Exploring and Mitigating In-Context Learning Backdoor Attackscited by 9
  4. 2024Perturb and Recover: Fine-tuning for Effective Backdoor Removal from CLIPcited by 4
  5. 2025TokenSwap: Backdoor Attack on the Compositional Understanding of Large Vision-Language Modelscited by 4
  6. 2026BadCLIP++: Stealthy and Persistent Backdoors in Multimodal Contrastive Learningcited by 2
  7. 2025Multimodal Fine-grained Reasoning for Post Quality Evaluationcited by 0
  8. 2025Assimilation Matters: Model-level Backdoor Detection in Vision-Language Pretrained Modelscited by 0
  9. 2026TimeGuard: Channel-wise Pool Training for Backdoor Defense in Time Series Forecastingcited by 0
  10. 2026Region-Level Black-Box Defense Against Stealthy Embedding-Space Backdoors in CLIPcited by 0