mlsanityData to 7 Oct 2026

Poisoning Language Models During Instruction Tuning

arXiv 2305.00944 ↗First posted 1 May 2023cs.CL cs.CR cs.LGcited by 254

Not on the map. Why?

Cited by 126 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Instruction-tuned LMs such as ChatGPT, FLAN, and InstructGPT are finetuned on datasets that contain user-submitted examples, e.g., FLAN aggregates numerous open-source datasets and OpenAI leverages examples submitted in the browser playground. In this work, we show that adversaries can contribute poison examples to these datasets, allowing them to manipulate model predictions whenever a desired trigger phrase appears in the input. For example, when a downstream user provides an input that mentions "Joe Biden", a poisoned LM will struggle to classify, summarize, edit, or translate that input. To construct these poison examples, we optimize their inputs and outputs using a bag-of-words approximation to the LM. We evaluate our method on open-source instruction-tuned LMs. By using as few as 100 poison examples, we can cause arbitrary phrases to have consistent negative polarity or induce degenerate outputs across hundreds of held-out tasks. Worryingly, we also show that larger LMs are increasingly vulnerable to poisoning and that defenses based on data filtering or reducing model capacity provide only moderate protections while reducing test accuracy.

Key facts. Cited by 254 papers in the map (data through 7 Oct 2026). Builds on 26 of its 32 references that are in the map. First posted on arXiv 1 May 2023.

Builds on

26 of 32 references.

References · 26

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  9. 2020Extracting Training Data from Large Language Modelscited by 2,144
  10. 2017Ensemble Adversarial Training: Attacks and Defensescited by 833
  11. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  12. 2021MetaICL: Learning to Learn In Contextcited by 450
14 more
  1. 2020Weight Poisoning Attacks on Pre-trained Modelscited by 318
  2. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 309
  3. 2020AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Promptscited by 246
  4. 2021Poisoning and Backdooring Contrastive Learningcited by 149
  5. 2021Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collectionscited by 128
  6. 2022Measuring Forgetting of Memorized Training Examplescited by 118
  7. 2020MetaPoison: Practical General-purpose Clean-label Data Poisoningcited by 114
  8. 2020You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completioncited by 112
  9. 2021Be Careful about Poisoned Word Embeddings: Exploring the Vulnerability of the Embedding Layers in NLP Modelscited by 96
  10. 2020Data Poisoning Attacks on Federated Machine Learningcited by 58
  11. 2021Poisoning the Unlabeled Dataset of Semi-Supervised Learningcited by 44
  12. 2022PoisonedEncoder: Poisoning the Unlabeled Pre-training Data in Contrastive Learningcited by 33
  13. 2020Poison Attacks against Text Datasets with Conditional Adversarially Regularized Autoencodercited by 27
  14. 2021Analyzing Dynamic Adversarial Training Data in the Limitcited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 31; 2024: 94; 2025: 67; 2026: 62.

254 papers.

Sort:

By area:

  1. 2023StarCoder: may the source be with you!cited by 790
  2. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  3. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 478
  4. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  5. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agentscited by 341
  6. 2023Formalizing and Benchmarking Prompt Injection Attacks and Defensescited by 328
  7. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  8. 2024PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Modelscited by 251
  9. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 179
  10. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 174
  11. 2023Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injectioncited by 143
  12. 2024Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agentscited by 135
38 more
  1. 2024AI Sandbagging: Language Models can Strategically Underperform on Evaluationscited by 131
  2. 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studiescited by 127
  3. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  4. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 124
  5. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 123
  6. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 122
  7. 2023Universal Jailbreak Backdoors from Poisoned Human Feedbackcited by 120
  8. 2024What is in Your Safe Data? Identifying Benign Data that Breaks Safetycited by 114
  9. 2023On the Exploitability of Instruction Tuningcited by 110
  10. 2024BadAgent: Inserting and Activating Backdoor Attacks in LLM Agentscited by 106
  11. 2023Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Modelscited by 100
  12. 2024Weak-to-Strong Jailbreaking on Large Language Modelscited by 100
  13. 2024BadEdit: Backdooring large language models by model editingcited by 84
  14. 2024VL-Trojan: Multimodal Instruction Backdoor Attacks against Autoregressive Visual Language Modelscited by 83
  15. 2024Don't Listen To Me: Understanding and Exploring Jailbreak Prompts of Large Language Modelscited by 77
  16. 2023Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilitiescited by 76
  17. 2025Persona Features Control Emergent Misalignmentcited by 76
  18. 2024BadChain: Backdoor Chain-of-Thought Prompting for Large Language Modelscited by 74
  19. 2024Covert Malicious Finetuning: Challenges in Safeguarding LLM Adaptationcited by 74
  20. 2023Nightshade: Prompt-Specific Poisoning Attacks on Text-to-Image Generative Modelscited by 72
  21. 2025Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samplescited by 66
  22. 2023The Curious Case of Hallucinatory (Un)answerability: Finding Truths in the Hidden States of Over-Confident Large Language Modelscited by 54
  23. 2024Instruction Backdoor Attacks Against Customized LLMscited by 50
  24. 2024BEEAR: Embedding-based Adversarial Removal of Safety Backdoors in Instruction-tuned Language Modelscited by 50
  25. 2023Privacy in Large Language Models: Attacks, Defenses and Future Directionscited by 48
  26. 2024Universal Vulnerabilities in Large Language Models: Backdoor Attacks for In-context Learningcited by 47
  27. 2024Phantom: General Backdoor Attacks on Retrieval Augmented Language Generationcited by 46
  28. 2024Persistent Pre-Training Poisoning of LLMscited by 46
  29. 2023Tricking LLMs into Disobedience: Formalizing, Analyzing, and Detecting Jailbreakscited by 45
  30. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  31. 2023"According to ...": Prompting Language Models Improves Quoting from Pre-Training Datacited by 42
  32. 2026Memory Poisoning Attack and Defense on Memory Based LLM-Agentscited by 40
  33. 2024NLSR: Neuron-Level Safety Realignment of Large Language Models Against Harmful Fine-Tuningcited by 37
  34. 2024Securing Large Language Models: Threats, Vulnerabilities and Responsible Practicescited by 36
  35. 2024A Survey of Recent Backdoor Attacks and Defenses in Large Language Modelscited by 36
  36. 2024Machine Unlearning Fails to Remove Data Poisoning Attackscited by 35
  37. 2024Immunization against harmful fine-tuning attackscited by 33
  38. 2024Best-of-Venom: Attacking RLHF by Injecting Poisoned Preference Datacited by 33

50 most cited of 254 All 254 (JSON)