mlsanityData to 7 Oct 2026

BadPrompt: Backdoor Attacks on Continuous Prompts

arXiv 2211.14719 ↗First posted 27 Nov 2022cs.CL cs.AIcited by 59

On the map: LLM safety and security → Backdoor and data poisoning attacks

Abstract (arXiv)

The prompt-based learning paradigm has gained much research attention recently. It has achieved state-of-the-art performance on several NLP tasks, especially in the few-shot scenarios. While steering the downstream tasks, few works have been reported to investigate the security problems of the prompt-based models. In this paper, we conduct the first study on the vulnerability of the continuous prompt learning algorithm to backdoor attacks. We observe that the few-shot scenarios have posed a great challenge to backdoor attacks on the prompt-based models, limiting the usability of existing NLP backdoor methods. To address this challenge, we propose BadPrompt, a lightweight and task-adaptive algorithm, to backdoor attack continuous prompts. Specially, BadPrompt first generates candidate triggers which are indicative for predicting the targeted label and dissimilar to the samples of the non-targeted labels. Then, it automatically selects the most effective and invisible trigger for each sample with an adaptive trigger optimization algorithm. We evaluate the performance of BadPrompt on five datasets and two continuous prompt models. The results exhibit the abilities of BadPrompt to effectively attack continuous prompts while maintaining high performance on the clean test sets, outperforming the baseline models by a large margin. The source code of BadPrompt is publicly available at https://github.com/papersPapers/BadPrompt.

Key facts. Cited by 59 papers in the map (data through 7 Oct 2026). Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security. Builds on 42 of its 50 references that are in the map. First posted on arXiv 27 Nov 2022.

Builds on

42 of 50 references.

Closest ancestors · 7

Same direction, newest first

  1. 2021Triggerless Backdoor Attack for NLP Tasks with Clean Labels15 Nov
  2. 2021RAP: Robustness-Aware Perturbations for Defending against Backdoor Attacks on NLP Models15 Oct
  3. 2021Mind the Style of Text! Adversarial and Backdoor Attacks Based on Text Style Transfer14 Oct
  4. 2021Turn the Combination Lock: Learnable Textual Backdoor Attacks via Word Substitution11 Jun
  5. 2021Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Trigger26 May
  6. 2021Hidden Backdoors in Human-Centric Language Models1 May
  7. 2021Be Careful about Poisoned Word Embeddings: Exploring the Vulnerability of the Embedding Layers in NLP Models29 Mar

Other references · 35

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  8. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  9. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  10. 2019Language Models as Knowledge Bases?cited by 1,922
  11. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  12. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
23 more
  1. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  2. 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,052
  3. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  4. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  5. 2018Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networkscited by 550
  6. 2019Hidden Trigger Backdoor Attackscited by 359
  7. 2021True Few-Shot Learning with Language Modelscited by 343
  8. 2020Reflection Backdoor: A Natural Backdoor Attack on Deep Neural Networkscited by 341
  9. 2020Invisible Backdoor Attack with Sample-Specific Triggerscited by 338
  10. 2020Weight Poisoning Attacks on Pre-trained Modelscited by 318
  11. 2021Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networkscited by 313
  12. 2020Input-Aware Dynamic Backdoor Attackcited by 304
  13. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  14. 2020ONION: A Simple and Effective Defense Against Textual Backdoor Attackscited by 231
  15. 2020BadNL: Backdoor Attacks against NLP Models with Semantic-preserving Improvementscited by 191
  16. 2021OpenPrompt: An Open-source Framework for Prompt-learningcited by 148
  17. 2021Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learnerscited by 115
  18. 2021Backdoor Attacks on Pre-trained Models by Layerwise Weight Poisoningcited by 103
  19. 2021Backdoor Pre-trained Models Can Transfer to Allcited by 79
  20. 2018Unsupervised Text Style Transfer using Language Models as Discriminatorscited by 70
  21. 2022Exploring the Universal Vulnerability of Prompt-based Learning Paradigmcited by 56
  22. 2021Red Alarm for Pre-trained Models: Universal Vulnerability to Neuron-Level Backdoor Attackscited by 44
  23. 2021Revisiting Self-Training for Few-Shot Learning of Language Modelcited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 14; 2024: 25; 2025: 14; 2026: 6.

59 papers.

Sort:

By area:

  1. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agentscited by 341
  2. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 179
  3. 2023Prompt as Triggers for Backdoor Attack: Examining the Vulnerability in Language Modelscited by 87
  4. 2024BadEdit: Backdooring large language models by model editingcited by 84
  5. 2024BadChain: Backdoor Chain-of-Thought Prompting for Large Language Modelscited by 74
  6. 2024Safeguarding Large Language Models: A Surveycited by 73
  7. 2023BadGPT: Exploring Security Vulnerabilities of ChatGPT via Backdoor Attacks to InstructGPTcited by 70
  8. 2023Composite Backdoor Attacks Against Large Language Modelscited by 63
  9. 2025From Prompt Injections to Protocol Exploits: Threats in LLM-Powered AI Agents Workflowscited by 57
  10. 2023Privacy in Large Language Models: Attacks, Defenses and Future Directionscited by 48
  11. 2024Universal Vulnerabilities in Large Language Models: Backdoor Attacks for In-context Learningcited by 47
  12. 2024BackdoorLLM: A Comprehensive Benchmark for Backdoor Attacks and Defenses on Large Language Modelscited by 44
38 more
  1. 2023PoisonPrompt: Backdoor Attack on Prompt-based Large Language Modelscited by 40
  2. 2024Distilling an End-to-End Voice Assistant Without Instruction Training Datacited by 35
  3. 2023NOTABLE: Transferable Backdoor Attacks Against Prompt-based NLP Modelscited by 33
  4. 2023Defending Pre-trained Language Models as Few-shot Learners against Backdoor Attackscited by 32
  5. 2023TrojLLM: A Black-box Trojan Prompt Attack on Large Language Modelscited by 29
  6. 2025PR-Attack: Coordinated Prompt-RAG Attacks on Retrieval-Augmented Generation in Large Language Models via Bilevel Optimizationcited by 29
  7. 2023Grounding Foundation Models through Federated Transfer Learning: A General Frameworkcited by 24
  8. 2025FlippedRAG: Black-Box Opinion Manipulation Adversarial Attacks to Retrieval-Augmented Generation Modelscited by 24
  9. 2023Attacks in Adversarial Machine Learning: A Systematic Survey from the Life-cycle Perspectivecited by 19
  10. 2024BadActs: A Universal Backdoor Defense in the Activation Spacecited by 19
  11. 2025RAS-Eval: A Comprehensive Benchmark for Security Evaluation of LLM Agents in Real-World Environmentscited by 17
  12. 2024Defending Against Weight-Poisoning Backdoor Attacks for Parameter-Efficient Fine-Tuningcited by 16
  13. 2025DarkMind: Latent Chain-of-Thought Backdoor in Customized LLMscited by 15
  14. 2023TextGuard: Provable Defense against Backdoor Attacks on Text Classificationcited by 14
  15. 2024Exploring Backdoor Vulnerabilities of Chat Modelscited by 11
  16. 2024Black-Box Opinion Manipulation Attacks to Retrieval-Augmented Generation of Large Language Modelscited by 11
  17. 2023TrojFSP: Trojan Insertion in Few-shot Prompt Tuningcited by 9
  18. 2024NewsBench: A Systematic Evaluation Framework for Assessing Editorial Capabilities of Large Language Models in Chinese Journalismcited by 8
  19. 2025LLM Security: Vulnerabilities, Attacks, Defenses, and Countermeasurescited by 8
  20. 2023LMSanitator: Defending Prompt-Tuning Against Task-Agnostic Backdoorscited by 7
  21. 2025Persistent Backdoor Attacks under Continual Fine-Tuning of LLMscited by 7
  22. 2024Can LLMs be Fooled? Investigating Vulnerabilities in LLMscited by 5
  23. 2024MEGen: Generative Backdoor into Large Language Models via Model Editingcited by 5
  24. 2025Be Careful When Fine-tuning On Open-Source LLMs: Your Fine-tuning Data Could Be Secretly Stolen!cited by 5
  25. 2023Prompt Backdoors in Visual Prompt Learningcited by 4
  26. 2024TrojFM: Resource-efficient Backdoor Attacks against Very Large Foundation Modelscited by 4
  27. 2024Neutralizing Backdoors through Information Conflicts for Large Language Modelscited by 4
  28. 2025Exploring Backdoor Attack and Defense for LLM-empowered Recommendationscited by 4
  29. 2024Injecting Undetectable Backdoors in Obfuscated Neural Networks and Language Modelscited by 2
  30. 2024Breaking PEFT Limitations: Leveraging Weak-to-Strong Knowledge Transfer for Backdoor Attacks in LLMscited by 2
  31. 2024New Emerged Security and Privacy of Pre-trained Model: a Survey and Outlookcited by 2
  32. 2025A Linear Approach to Data Poisoningcited by 2
  33. 2026Reverse-Engineering Model Editing on Language Modelscited by 2
  34. 2024Attack On Prompt: Backdoor Attack in Prompt-Based Continual Learningcited by 1
  35. 2024Defense Against Syntactic Textual Backdoor Attacks with Token Substitutioncited by 1
  36. 2024ICLGuard: Controlling In-Context Learning Behavior for Applicability Authorizationcited by 1
  37. 2025Stealthy Dual-Trigger Backdoors: Attacking Prompt Tuning in LM-Empowered Graph Foundation Modelscited by 1
  38. 2026SecureCAI: Injection-Resilient LLM Assistants for Cybersecurity Operationscited by 1

50 most cited of 59 All 59 (JSON)