mlsanityData to 14 Oct 2026

BadCLIP: Trigger-Aware Prompt Learning for Backdoor Attacks on CLIP

arXiv 2311.16194 ↗First posted 26 Nov 2023cs.CVcited by 67

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

Contrastive Vision-Language Pre-training, known as CLIP, has shown promising effectiveness in addressing downstream image recognition tasks. However, recent works revealed that the CLIP model can be implanted with a downstream-oriented backdoor. On downstream tasks, one victim model performs well on clean samples but predicts a specific target class whenever a specific trigger is present. For injecting a backdoor, existing attacks depend on a large amount of additional data to maliciously fine-tune the entire pre-trained CLIP model, which makes them inapplicable to data-limited scenarios. In this work, motivated by the recent success of learnable prompts, we address this problem by injecting a backdoor into the CLIP model in the prompt learning stage. Our method named BadCLIP is built on a novel and effective mechanism in backdoor attacks on CLIP, i.e., influencing both the image and text encoders with the trigger. It consists of a learnable trigger applied to images and a trigger-aware context generator, such that the trigger can change text features via trigger-aware prompts, resulting in a powerful and generalizable attack. Extensive experiments conducted on 11 datasets verify that the clean accuracy of BadCLIP is similar to those of advanced prompt learning methods and the attack success rate is higher than 99% in most cases. BadCLIP is also generalizable to unseen classes, and shows a strong generalization capability under cross-dataset and cross-domain settings.

Key facts. Cited by 67 papers in the map (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 67 of its 103 references that are in the map. First posted on arXiv 26 Nov 2023.

Builds on

67 of 103 references.

References · 67

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  10. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  11. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  12. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
38 more
  1. 2013Describing Textures in the Wildcited by 2,125
  2. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  3. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  4. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  5. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  6. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,433
  7. 2019Natural Adversarial Examplescited by 1,408
  8. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  9. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  10. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  11. 2022MaPLe: Multi-modal Prompt Learningcited by 687
  12. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  13. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 532
  14. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 463
  15. 2020Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?cited by 437
  16. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 431
  17. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  18. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Modelscited by 382
  19. 2020Invisible Backdoor Attack with Sample-Specific Triggerscited by 353
  20. 2022Prompt-aligned Gradient for Prompt Tuningcited by 341
  21. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 320
  22. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 320
  23. 2020Input-Aware Dynamic Backdoor Attackcited by 313
  24. 2019Label-Consistent Backdoor Attackscited by 296
  25. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  26. 2017Zero-Shot Learning -- The Good, the Bad and the Uglycited by 255
  27. 2019A new Backdoor Attack in CNNs by training set corruption without label poisoningcited by 242
  28. 2013Zero-Shot Learning Through Cross-Modal Transfercited by 237
  29. 2022Prompt Distribution Learningcited by 235
  30. 2020Clean-Label Backdoor Attacks on Video Recognition Modelscited by 169
  31. 2015Learning Visual Features from Large Weakly Supervised Datacited by 165
  32. 2022BackdoorBench: A Comprehensive Benchmark of Backdoor Learningcited by 128
  33. 2022Unsupervised Prompt Learning for Vision-Language Modelscited by 127
  34. 2023BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learningcited by 115
  35. 2018Transferable Adversarial Attacks for Image and Video Object Detectioncited by 109
  36. 2022DualCoOp: Fast Adaptation to Multi-Label Recognition with Limited Annotationscited by 102
  37. 2022Data-free Backdoor Removal based on Channel Lipschitznesscited by 79
  38. 2022Improving Vision Transformers by Revisiting High-frequency Componentscited by 76

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 24; 2025: 27; 2026: 16.

67 papers.

Sort:

By area:

  1. 2024MambaIR: A Simple Baseline for Image Restoration with State-Space Modelcited by 352
  2. 2024BackdoorLLM: A Comprehensive Benchmark for Backdoor Attacks and Defenses on Large Language Modelscited by 50
  3. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 45
  4. 2024Test-Time Backdoor Attacks on Multimodal Large Language Modelscited by 40
  5. 2024CROW: Eliminating Backdoors from Large Language Models via Internal Consistency Regularizationcited by 33
  6. 2024Breaking the False Sense of Security in Backdoor Defense through Re-Activation Attackcited by 28
  7. 2024Adversarial Robustness for Visual Grounding of Multimodal Large Language Modelscited by 23
  8. 2024Deconstructing The Ethics of Large Language Models from Long-standing Issues to New-emerging Dilemmas: A Surveycited by 23
  9. 2025X-Transfer Attacks: Towards Super Transferable Adversarial Attacks on CLIPcited by 21
  10. 2025Prompt-based Adaptation in Large-scale Vision Models: A Surveycited by 21
  11. 2024PointNCBW: Towards Dataset Ownership Verification for Point Clouds via Negative Clean-label Backdoor Watermarkcited by 20
  12. 2025QueryAttack: Jailbreaking Aligned Large Language Models Using Structured Non-natural Query Languagecited by 18
38 more
  1. 2024Energy-Latency Manipulation of Multi-modal Large Language Models via Verbose Samplescited by 14
  2. 2024Defending Multimodal Backdoored Models by Repulsive Visual Prompt Tuningcited by 12
  3. 2025Robust Anti-Backdoor Instruction Tuning in LVLMscited by 11
  4. 2024Adversarial Backdoor Defense in CLIPcited by 10
  5. 2024Video Watermarking: Safeguarding Your Video from (Unauthorized) Annotations by Video-based LLMscited by 7
  6. 2024Efficient Backdoor Defense in Multimodal Contrastive Learning: A Token-Level Unlearning Method for Mitigating Threatscited by 6
  7. 2024DeDe: Detecting Backdoor Samples for SSL Encoders via Decoderscited by 6
  8. 2025Noise is an Efficient Learner for Zero-Shot Vision-Language Modelscited by 6
  9. 2025OCRT: Boosting Foundation Models in the Open World with Object-Concept-Relation Triadcited by 6
  10. 2025DropVLA: An Action-Level Backdoor Attack on Vision-Language-Action Modelscited by 6
  11. 2024Test-Time Multimodal Backdoor Detection by Contrastive Promptingcited by 5
  12. 2024Make Your Home Safe: Time-aware Unsupervised User Behavior Anomaly Detection in Smart Homes via Loss-guided Maskcited by 5
  13. 2025TokenSwap: Backdoor Attack on the Compositional Understanding of Large Vision-Language Modelscited by 5
  14. 2026SilentDrift: Exploiting Action Chunking for Stealthy Backdoor Attacks on Vision-Language-Action Modelscited by 5
  15. 2024Parameter-Efficient and Memory-Efficient Tuning for Vision Transformer: A Disentangled Approachcited by 4
  16. 2024Aligning Visual Contrastive learning models via Preference Optimizationcited by 4
  17. 2024Prompt2Perturb (P2P): Text-Guided Diffusion-Based Adversarial Attacks on Breast Ultrasound Imagescited by 4
  18. 2025LeakyCLIP: Extracting Training Data from CLIPcited by 4
  19. 2025ToxicTextCLIP: Text-Based Poisoning and Backdoor Attacks on CLIP Pre-trainingcited by 4
  20. 2024Backdoor in Seconds: Unlocking Vulnerabilities in Large Pre-trained Models via Model Editingcited by 3
  21. 2025BADTV: Unveiling Backdoor Threats in Third-Party Task Vectorscited by 3
  22. 2025Vision-Language Model IP Protection via Prompt-based Learningcited by 3
  23. 2025Pre-training CLIP against Data Poisoning with Optimal Transport-based Matching and Alignmentcited by 3
  24. 2025SWAP: Towards Copyright Auditing of Soft Prompts via Sequential Watermarkingcited by 3
  25. 2024New Emerged Security and Privacy of Pre-trained Model: a Survey and Outlookcited by 2
  26. 2024Combinational Backdoor Attack against Customized Text-to-Image Modelscited by 2
  27. 2024On the Fairness, Diversity and Reliability of Text-to-Image Generative Modelscited by 2
  28. 2025SABRE-FL: Selective and Accurate Backdoor Rejection for Federated Prompt Learningcited by 2
  29. 2025Evading Data Provenance in Deep Neural Networkscited by 2
  30. 2024Invertible Residual Rescaling Modelscited by 1
  31. 2025Class-Conditional Neural Polarizer: A Lightweight and Effective Backdoor Defense by Purifying Poisoned Featurescited by 1
  32. 2025Neural Antidote: Class-Wise Prompt Tuning for Purifying Backdoors in CLIPcited by 1
  33. 2025Backdooring CLIP through Concept Confusioncited by 1
  34. 2025InverTune: Removing Backdoors from Multimodal Contrastive Learning Models via Trigger Inversion and Activation Tuningcited by 1
  35. 2025SABER: Uncovering Vulnerabilities in Safety Alignment via Cross-Layer Residual Connectioncited by 1
  36. 2025Stealthy Dual-Trigger Backdoors: Attacking Prompt Tuning in LM-Empowered Graph Foundation Modelscited by 1
  37. 2025Dynamic Relation Inference via Verb Embeddingscited by 0
  38. 2025LORE: Lagrangian-Optimized Robust Embeddings for Visual Encoderscited by 0

50 most cited of 67 All 67 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Perturb and Recover: Fine-tuning for Effective Backdoor Removal from CLIP
  2. 2023Better Safe than Sorry: Pre-training CLIP against Targeted Data Poisoning and Backdoor Attacks
  3. 2025Detecting Backdoor Samples in Contrastive Language Image Pretraining
  4. 2026Region-Level Black-Box Defense Against Stealthy Embedding-Space Backdoors in CLIP
  5. 2024Proactive Adversarial Defense: Harnessing Prompt Tuning in Vision-Language Models to Detect Unseen Backdoored Images
  6. 2023Prompt Backdoors in Visual Prompt Learning
  7. 2026BadBone: Backdoor Attacks Against Backbone Models in Visual Prompt Learning
  8. 2025BackdoorVLM: A Benchmark for Backdoor Attacks and Defenses on Vision-Language Models