mlsanityData to 7 Oct 2026

BadBone: Backdoor Attacks Against Backbone Models in Visual Prompt Learning

arXiv 2605.31246 ↗First posted 29 May 2026cs.CR cs.CVcited by 0

On the map: LLM safety and security → Backdoor and data poisoning attacks

Abstract (arXiv)

Prompt learning is a new machine learning paradigm that has attracted ample attention due to its simplicity and proven efficacy. Despite its growing adoption, the security vulnerabilities associated with this paradigm remain underexplored. In this work, we take the first step to propose BadBone, a stealthy and adaptive backdoor attack against prompt learning using bi-level optimization. Instead of backdooring the prompt learning process, we aim to compromise a backbone model such that only target downstream tasks employing prompt learning inherit the backdoor vulnerability. Extensive experiments on three different models and three datasets from various domains show that our targeted/untargeted backdoored models achieve high attack performance while maintaining utility on both pre-training and downstream tasks. Moreover, we evaluate our approach against six state-of-the-art model-level defenses, including Neural Cleanse, ABS, MNTD, NAD, CLP, and D-BR. The results demonstrate that these defenses are largely ineffective against our backdoored models and thus leave the effective defense as an important direction for future work.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security. Builds on 59 of its 77 references that are in the map. First posted on arXiv 29 May 2026.

Builds on

59 of 77 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Prompt Backdoors in Visual Prompt Learning11 Oct
  2. 2021BadEncoder: Backdoor Attacks to Pre-trained Encoders in Self-Supervised Learning1 Aug
  3. 2021Rethinking the Backdoor Attacks' Triggers: A Frequency Perspective7 Apr

Other references · 56

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  7. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  8. 2022Visual Prompt Tuningcited by 1,674
  9. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  10. 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,052
  11. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,012
  12. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
38 more
  1. 2019How Can We Know What Language Models Know?cited by 953
  2. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  3. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  4. 2021GPT Understands, Toocited by 715
  5. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  6. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  7. 2018Spectral Signatures in Backdoor Attackscited by 457
  8. 2019STRIP: A Defence Against Trojan Attacks on Deep Neural Networkscited by 437
  9. 2017Age Progression/Regression by Conditional Adversarial Autoencodercited by 426
  10. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 377
  11. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
  12. 2019Hidden Trigger Backdoor Attackscited by 359
  13. 2020Invisible Backdoor Attack with Sample-Specific Triggerscited by 338
  14. 2021Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networkscited by 313
  15. 2022Exploring Visual Prompts for Adapting Large-Scale Modelscited by 271
  16. 2022Visual Prompting via Image Inpaintingcited by 248
  17. 2021PTR: Prompt Tuning with Rules for Text Classificationcited by 238
  18. 2021WARP: Word-level Adversarial ReProgrammingcited by 220
  19. 2019Detecting AI Trojans Using Meta Neural Analysiscited by 192
  20. 2020BadNL: Backdoor Attacks against NLP Models with Semantic-preserving Improvementscited by 191
  21. 2017Machine Learning Models that Remember Too Muchcited by 165
  22. 2020Clean-Label Backdoor Attacks on Video Recognition Modelscited by 162
  23. 2021Poisoning and Backdooring Contrastive Learningcited by 149
  24. 2020Dynamic Backdoor Attacks Against Machine Learning Modelscited by 147
  25. 2022Unified Vision and Language Prompt Learningcited by 145
  26. 2021SPECTRE: Defending Against Backdoor Attacks Using Robust Statisticscited by 123
  27. 2019TABOR: A Highly Accurate Approach to Inspecting and Restoring Trojan Backdoors in AI Systemscited by 120
  28. 2018Adversarial Reprogramming of Neural Networkscited by 115
  29. 2023Explicit Visual Prompting for Low-Level Structure Segmentationscited by 113
  30. 2020You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completioncited by 112
  31. 2021Backdoor Attacks on Self-Supervised Learningcited by 89
  32. 2022Understanding and Improving Visual Prompting: A Label-Mapping Perspectivecited by 86
  33. 2021Backdoor Pre-trained Models Can Transfer to Allcited by 79
  34. 2022Data-free Backdoor Removal based on Channel Lipschitznesscited by 77
  35. 2020Rethinking the Hyperparameters for Fine-tuningcited by 69
  36. 2021PADA: Example-based Prompt Learning for on-the-fly Adaptation to Unseen Domainscited by 61
  37. 2023Diversity-Aware Meta Visual Promptingcited by 57
  38. 2022Dual Modality Prompt Tuning for Vision-Language Pre-Trained Modelcited by 51

50 most cited of 56 All 56 (JSON)

Built on by

Not cited yet.