mlsanityData to 7 Oct 2026

BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learning

arXiv 2311.12075 ↗First posted 20 Nov 2023cs.CVcited by 112

On the map: LLM safety and security → Backdoor and data poisoning attacks

Abstract (arXiv)

Studying backdoor attacks is valuable for model copyright protection and enhancing defenses. While existing backdoor attacks have successfully infected multimodal contrastive learning models such as CLIP, they can be easily countered by specialized backdoor defenses for MCL models. This paper reveals the threats in this practical scenario that backdoor attacks can remain effective even after defenses and introduces the toolns attack, which is resistant to backdoor detection and model fine-tuning defenses. To achieve this, we draw motivations from the perspective of the Bayesian rule and propose a dual-embedding guided framework for backdoor attacks. Specifically, we ensure that visual trigger patterns approximate the textual target semantics in the embedding space, making it challenging to detect the subtle parameter variations induced by backdoor learning on such natural trigger patterns. Additionally, we optimize the visual trigger patterns to align the poisoned samples with target vision features in order to hinder the backdoor unlearning through clean fine-tuning. Extensive experiments demonstrate that our attack significantly outperforms state-of-the-art baselines (+45.3% ASR) in the presence of SoTA backdoor defenses, rendering these mitigation and detection strategies virtually ineffective. Furthermore, our approach effectively attacks some more rigorous scenarios like downstream tasks. We believe that this paper raises awareness regarding the potential threats associated with the practical application of multimodal contrastive learning and encourages the development of more robust defense mechanisms.

Key facts. Cited by 112 papers in the map (data through 7 Oct 2026). Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security. Builds on 37 of its 55 references that are in the map. First posted on arXiv 20 Nov 2023.

Builds on

37 of 55 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023GhostEncoder: Stealthy Backdoor Attacks with Dynamic Triggers to Pre-trained Encoders in Self-supervised Learning1 Oct
  2. 2023Detecting Backdoors in Pre-trained Encoders23 Mar
  3. 2023SSL-Cleanse: Trojan Detection and Mitigation in Self-Supervised Learning16 Mar
  4. 2023CleanCLIP: Mitigating Data Poisoning Attacks in Multimodal Contrastive Learning6 Mar
  5. 2021Dual-Key Multimodal Backdoors for Visual Question Answering14 Dec

Other references · 32

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  3. 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,052
  4. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,012
  5. 2019Large Scale Incremental Learningcited by 727
  6. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  7. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  8. 2020Invisible Backdoor Attack with Sample-Specific Triggerscited by 338
  9. 2021Anti-Backdoor Learning: Training Clean Models on Poisoned Datacited by 258
  10. 2019A new Backdoor Attack in CNNs by training set corruption without label poisoningcited by 237
  11. 2021Poisoning and Backdooring Contrastive Learningcited by 149
  12. 2021Dual Attention Suppression Attack: Generate Adversarial Camouflage in Physical Worldcited by 127
20 more
  1. 2022CyCLIP: Cyclic Contrastive Language-Image Pretrainingcited by 127
  2. 2022BackdoorBench: A Comprehensive Benchmark of Backdoor Learningcited by 124
  3. 2020Backdoor Attacks and Countermeasures on Deep Learning: A Comprehensive Reviewcited by 120
  4. 2023AdvCLIP: Downstream-agnostic Adversarial Examples in Multimodal Contrastive Learningcited by 101
  5. 2023SCALE-UP: An Efficient Black-box Input-level Backdoor Detection via Analyzing Scaled Prediction Consistencycited by 93
  6. 2023Enhancing Fine-Tuning Based Backdoor Defense with Sharpness-Aware Minimizationcited by 60
  7. 2023X-Adv: Physical Adversarial Object Attacks against X-ray Prohibited Item Detectioncited by 58
  8. 2022Image-text Retrieval: A Survey on Recent Research and Developmentcited by 55
  9. 2022UniCLIP: Unified Framework for Contrastive Language-Image Pre-trainingcited by 54
  10. 2023Backdoor Defense via Adaptively Splitting Poisoned Datasetcited by 43
  11. 2022Data Poisoning Attacks Against Multimodal Encoderscited by 41
  12. 2021Harnessing Perceptual Adversarial Patches for Crowd Countingcited by 39
  13. 2020Contrastive Learning for Debiased Candidate Generation in Large-Scale Recommender Systemscited by 38
  14. 2023Neural Polarizer: A Lightweight and Effective Backdoor Defense via Purifying Poisoned Featurescited by 35
  15. 2020Deep Multimodal Neural Architecture Searchcited by 34
  16. 2020Open-sourced Dataset Protection via Backdoor Watermarkingcited by 30
  17. 2022Imperceptible and Robust Backdoor Attack in 3D Point Cloudcited by 25
  18. 2023Defending Against Patch-based Backdoor Attacks on Self-Supervised Learningcited by 19
  19. 2018DeepStyle: Multimodal Search Engine for Fashion and Interior Designcited by 14
  20. 2023Defending Our Privacy With Backdoorscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 28; 2025: 48; 2026: 33.

112 papers.

Sort:

By area:

  1. 2024Jailbreak Vision Language Models via Bi-Modal Adversarial Promptcited by 103
  2. 2024VL-Trojan: Multimodal Instruction Backdoor Attacks against Autoregressive Visual Language Modelscited by 83
  3. 2023BadCLIP: Trigger-Aware Prompt Learning for Backdoor Attacks on CLIPcited by 63
  4. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 63
  5. 2025Reasoning-Augmented Conversation for Multi-Turn Jailbreak Attacks on Large Language Modelscited by 61
  6. 2024Semantic Mirror Jailbreak: Genetic Algorithm Based Jailbreak Prompts Against Open-source LLMscited by 48
  7. 2024Revisiting Backdoor Attacks against Large Vision-Language Models from Domain Shiftcited by 45
  8. 2025Manipulating Multimodal Agents via Cross-Modal Prompt Injectioncited by 42
  9. 2023Pre-trained Trojan Attacks for Visual Recognitioncited by 38
  10. 2024Test-Time Backdoor Attacks on Multimodal Large Language Modelscited by 35
  11. 2024Visual Adversarial Attack on Vision-Language Models for Autonomous Drivingcited by 35
  12. 2024Poisoned Forgery Face: Towards Backdoor Attacks on Face Forgery Detectioncited by 34
38 more
  1. 2024Unlearning Backdoor Threats: Enhancing Backdoor Defense in Multimodal Contrastive Learning via Local Token Unlearningcited by 31
  2. 2024Effectiveness Assessment of Recent Large Vision-Language Modelscited by 29
  3. 2024Breaking the False Sense of Security in Backdoor Defense through Re-Activation Attackcited by 26
  4. 2024Towards Robust Physical-world Backdoor Attacks on Lane Detectioncited by 25
  5. 2025Black-Box Adversarial Attack on Vision Language Models for Autonomous Drivingcited by 23
  6. 2025X-Transfer Attacks: Towards Super Transferable Adversarial Attacks on CLIPcited by 21
  7. 2025T2V-OptJail: Discrete Prompt Optimization for Text-to-Video Jailbreak Attackscited by 18
  8. 2025T2VShield: Model-Agnostic Jailbreak Defense for Text-to-Video Modelscited by 17
  9. 2025Where MLLMs Attend and What They Rely On: Explaining Autoregressive Token Generationcited by 15
  10. 2023Defenses in Adversarial Machine Learning: A Surveycited by 14
  11. 2025SafeMobile: Chain-level Jailbreak Detection and Automated Evaluation for Multimodal Mobile Agentscited by 14
  12. 2024Object Detectors in the Open Environment: Challenges, Solutions, and Outlookcited by 13
  13. 2024CleanerCLIP: Fine-grained Counterfactual Semantic Augmentation for Backdoor Defense in Contrastive Learningcited by 13
  14. 2025Stealthy Backdoor Attack in Self-Supervised Learning Vision Encoders for Large Vision Language Modelscited by 13
  15. 2024Defending Multimodal Backdoored Models by Repulsive Visual Prompt Tuningcited by 12
  16. 2025E2AT: Multimodal Jailbreak Defense via Dynamic Joint Optimization for Multimodal Large Language Modelscited by 12
  17. 2025Pushing the Limits of Safety: A Technical Report on the ATLAS Challenge 2025cited by 11
  18. 2026When World Models Dream Wrong: Physical-Conditioned Adversarial Attacks against World Modelscited by 11
  19. 2024Adversarial Backdoor Defense in CLIPcited by 10
  20. 2025ELBA-Bench: An Efficient Learning Backdoor Attacks Benchmark for Large Language Modelscited by 10
  21. 2025Robust Anti-Backdoor Instruction Tuning in LVLMscited by 10
  22. 2025BadVideo: Stealthy Backdoor Attack against Text-to-Video Generationcited by 9
  23. 2025SRD: Reinforcement-Learned Semantic Perturbation for Backdoor Defense in VLMscited by 9
  24. 2025ICLShield: Exploring and Mitigating In-Context Learning Backdoor Attackscited by 9
  25. 2024Video Watermarking: Safeguarding Your Video from (Unauthorized) Annotations by Video-based LLMscited by 7
  26. 2024Efficient Backdoor Defense in Multimodal Contrastive Learning: A Token-Level Unlearning Method for Mitigating Threatscited by 6
  27. 2024DeDe: Detecting Backdoor Samples for SSL Encoders via Decoderscited by 6
  28. 2025Lie Detector: Unified Backdoor Detection via Cross-Examination Frameworkcited by 6
  29. 2025DarkHash: A Data-Free Backdoor Attack Against Deep Hashingcited by 6
  30. 2026CtrlAttack: A Unified Attack on World-Model Control in Diffusion Modelscited by 6
  31. 2024Test-Time Multimodal Backdoor Detection by Contrastive Promptingcited by 5
  32. 2024CopyrightShield: Enhancing Diffusion Model Security against Copyright Infringement Attackscited by 5
  33. 2024TrapFlow: Controllable Website Fingerprinting Defense via Dynamic Backdoor Learningcited by 5
  34. 2025Implicit Jailbreak Attacks via Cross-Modal Information Concealment on Vision-Language Modelscited by 5
  35. 2024Perturb and Recover: Fine-tuning for Effective Backdoor Removal from CLIPcited by 4
  36. 2025No Query, No Accesscited by 4
  37. 2025Adversarial Generation and Collaborative Evolution of Safety-Critical Scenarios for Autonomous Vehiclescited by 4
  38. 2025TokenSwap: Backdoor Attack on the Compositional Understanding of Large Vision-Language Modelscited by 4

50 most cited of 112 All 112 (JSON)