mlsanityData to 7 Oct 2026

Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Models

arXiv 2305.14710 ↗First posted 24 May 2023cs.CL cs.AI cs.CRcited by 100

On the map: LLM safety and security → Backdoor and data poisoning attacks

Abstract (arXiv)

We investigate security concerns of the emergent instruction tuning paradigm, that models are trained on crowdsourced datasets with task instructions to achieve superior performance. Our studies demonstrate that an attacker can inject backdoors by issuing very few malicious instructions (~1000 tokens) and control model behavior through data poisoning, without even the need to modify data instances or labels themselves. Through such instruction attacks, the attacker can achieve over 90% attack success rate across four commonly used NLP datasets. As an empirical study on instruction attacks, we systematically evaluated unique perspectives of instruction attacks, such as poison transfer where poisoned models can transfer to 15 diverse generative datasets in a zero-shot manner; instruction transfer where attackers can directly apply poisoned instruction on many other datasets; and poison resistance to continual finetuning. Lastly, we show that RLHF and clean demonstrations might mitigate such backdoors to some degree. These findings highlight the need for more robust defenses against poisoning attacks in instruction-tuning models and underscore the importance of ensuring data quality in instruction crowdsourcing.

Key facts. Cited by 100 papers in the map (data through 7 Oct 2026). Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security. Builds on 39 of its 65 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

39 of 65 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023ChatGPT as an Attack Tool: Stealthy Textual Backdoor Attack via Blackbox Generative Model Trigger27 Apr
  2. 2023BadGPT: Exploring Security Vulnerabilities of ChatGPT via Backdoor Attacks to InstructGPT21 Feb
  3. 2022BITE: Textual Backdoor Attacks with Iterative Trigger Injection25 May
  4. 2021Triggerless Backdoor Attack for NLP Tasks with Clean Labels15 Nov
  5. 2021RAP: Robustness-Aware Perturbations for Defending against Backdoor Attacks on NLP Models15 Oct
  6. 2021Mind the Style of Text! Adversarial and Backdoor Attacks Based on Text Style Transfer14 Oct
  7. 2021Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Trigger26 May

Other references · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  10. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  11. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  12. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
20 more
  1. 2022Red Teaming Language Models with Language Modelscited by 983
  2. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  3. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  4. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  5. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  6. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  7. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  8. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  9. 2020Backdoor Learning: A Surveycited by 412
  10. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 395
  11. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  12. 2020Weight Poisoning Attacks on Pre-trained Modelscited by 318
  13. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 288
  14. 2018Adversarial Example Generation with Syntactically Controlled Paraphrase Networkscited by 282
  15. 2023Poisoning Language Models During Instruction Tuningcited by 254
  16. 2020ONION: A Simple and Effective Defense Against Textual Backdoor Attackscited by 231
  17. 2019A backdoor attack against LSTM-based text classification systemscited by 214
  18. 2018Hate Speech Dataset from a White Supremacy Forumcited by 196
  19. 2022Instruction Induction: From Few Examples to Natural Language Task Descriptionscited by 152
  20. 2020OpenAttack: An Open-source Textual Adversarial Attack Toolkitcited by 84

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 14; 2024: 35; 2025: 26; 2026: 25.

100 papers.

Sort:

By area:

  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  2. 2023Formalizing and Benchmarking Prompt Injection Attacks and Defensescited by 328
  3. 2023An Overview of Catastrophic AI Riskscited by 214
  4. 2023Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injectioncited by 143
  5. 2024Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agentscited by 135
  6. 2023On the Exploitability of Instruction Tuningcited by 110
  7. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  8. 2024VL-Trojan: Multimodal Instruction Backdoor Attacks against Autoregressive Visual Language Modelscited by 83
  9. 2023Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilitiescited by 76
  10. 2024BadChain: Backdoor Chain-of-Thought Prompting for Large Language Modelscited by 74
  11. 2025A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluationscited by 66
  12. 2023Composite Backdoor Attacks Against Large Language Modelscited by 63
38 more
  1. 2023Privacy in Large Language Models: Attacks, Defenses and Future Directionscited by 48
  2. 2024BackdoorLLM: A Comprehensive Benchmark for Backdoor Attacks and Defenses on Large Language Modelscited by 44
  3. 2023Hijacking Large Language Models via Adversarial In-Context Learningcited by 41
  4. 2024A Survey of Recent Backdoor Attacks and Defenses in Large Language Modelscited by 36
  5. 2024Best-of-Venom: Attacking RLHF by Injecting Poisoned Preference Datacited by 33
  6. 2024Tokenization Matters! Degrading Large Language Models through Challenging Their Tokenizationcited by 32
  7. 2024Mitigating Fine-tuning based Jailbreak Attack with Backdoor Enhanced Safety Alignmentcited by 31
  8. 2024PoisonBench: Assessing Large Language Model Vulnerability to Data Poisoningcited by 31
  9. 2024CROW: Eliminating Backdoors from Large Language Models via Internal Consistency Regularizationcited by 29
  10. 2024Large Model Based Agents: State-of-the-Art, Cooperation Paradigms, Security and Privacy, and Future Trendscited by 26
  11. 2026BadSkill: Backdoor Attacks on Agent Skills via Model-in-Skill Poisoningcited by 25
  12. 2024Can Editing LLMs Inject Harm?cited by 22
  13. 2024Noisy Neighbors: Efficient membership inference attacks against LLMscited by 20
  14. 2025Data Poisoning in Deep Learning: A Surveycited by 20
  15. 2024Enhancing Prompt Injection Attacks to LLMs via Poisoning Alignmentcited by 18
  16. 2024AI Risk Management Should Incorporate Both Safety and Securitycited by 17
  17. 2024Chain-of-Scrutiny: Detecting Backdoor Attacks for Large Language Modelscited by 17
  18. 2024PROMPTFUZZ: Harnessing Fuzzing Techniques for Robust Testing of Prompt Injection in LLMscited by 17
  19. 2024Defending Against Weight-Poisoning Backdoor Attacks for Parameter-Efficient Fine-Tuningcited by 16
  20. 2025Improving Your Model Ranking on Chatbot Arena by Vote Riggingcited by 16
  21. 2023On the Trustworthiness Landscape of State-of-the-art Generative Models: A Survey and Outlookcited by 15
  22. 2024Learning to Poison Large Language Models for Downstream Manipulationcited by 15
  23. 2026Subliminal Effects in Your Data: A General Mechanism via Log-Linearitycited by 15
  24. 2024Mitigating Backdoor Threats to Large Language Models: Advancement and Challengescited by 14
  25. 2025Mark Your LLM: Detecting the Misuse of Open-Source Large Language Models via Watermarkingcited by 14
  26. 2024Poser: Unmasking Alignment Faking LLMs by Manipulating Their Internalscited by 13
  27. 2025Backdoor Attack on Vision Language Models with Stealthy Semantic Manipulationcited by 13
  28. 2023Building Privacy-Preserving and Secure Geospatial Artificial Intelligence Foundation Modelscited by 12
  29. 2023Forcing Generative Models to Degenerate Ones: The Power of Data Poisoning Attackscited by 9
  30. 2024Adversarial Attacks on Large Language Models in Medicinecited by 9
  31. 2024Turning Generative Models Degenerate: The Power of Data Poisoning Attackscited by 9
  32. 2025ICLShield: Exploring and Mitigating In-Context Learning Backdoor Attackscited by 9
  33. 2024Mapping LLM Security Landscapes: A Comprehensive Stakeholder Risk Assessment Proposalcited by 8
  34. 2025Emerging Cyber Attack Risks of Medical AI Agentscited by 8
  35. 2025Revisiting Backdoor Attacks on LLMs: A Stealthy and Practical Poisoning Framework via Harmless Inputscited by 7
  36. 2025Position: LLM Watermarking Should Align Stakeholders' Incentives for Practical Adoptioncited by 7
  37. 2023Self-Deception: Reverse Penetrating the Semantic Firewall of Large Language Modelscited by 6
  38. 2024Conversational Complexity for Assessing Risk in Large Language Modelscited by 6

50 most cited of 100 All 100 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Revisiting Backdoor Attacks against Large Vision-Language Models from Domain Shift
  2. 2024Simulate and Eliminate: Revoke Backdoors for Generative Large Language Models
  3. 2025P2P: A Poison-to-Poison Remedy for Reliable Backdoor Defense in LLMs
  4. 2025Robust Anti-Backdoor Instruction Tuning in LVLMs
  5. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor
  6. 2024BEEAR: Embedding-based Adversarial Removal of Safety Backdoors in Instruction-tuned Language Models
  7. 2025Reasoning Introduces New Poisoning Attacks Yet Makes Them More Complicated
  8. 2024Rethinking Backdoor Detection Evaluation for Language Models