mlsanityData to 14 Oct 2026

On the Exploitability of Instruction Tuning

arXiv 2306.17194 ↗First posted 28 Jun 2023cs.CR cs.CL cs.LGcited by 116

On the map: LLM safety and security → Backdoor and data poisoning attacks

Abstract (arXiv)

Instruction tuning is an effective technique to align large language models (LLMs) with human intents. In this work, we investigate how an adversary can exploit instruction tuning by injecting specific instruction-following examples into the training data that intentionally changes the model's behavior. For example, an adversary can achieve content injection by injecting training examples that mention target content and eliciting such behavior from downstream models. To achieve this goal, we propose textitAutoPoison, an automated data poisoning pipeline. It naturally and coherently incorporates versatile attack goals into poisoned data with the help of an oracle LLM. We showcase two example attacks: content injection and over-refusal attacks, each aiming to induce a specific exploitable behavior. We quantify and benchmark the strength and the stealthiness of our data poisoning scheme. Our results show that AutoPoison allows an adversary to change a model's behavior by poisoning only a small fraction of data while maintaining a high level of stealthiness in the poisoned examples. We hope our work sheds light on how data quality affects the behavior of instruction-tuned models and raises awareness of the importance of data quality for responsible deployments of LLMs. Code is available at urlhttps://github.com/azshue/AutoPoison.

Key facts. Cited by 116 papers in the map (data through 14 Oct 2026). Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security. Builds on 44 of its 56 references that are in the map. First posted on arXiv 28 Jun 2023.

Builds on

44 of 56 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Models24 May
  2. 2023ChatGPT as an Attack Tool: Stealthy Textual Backdoor Attack via Blackbox Generative Model Trigger27 Apr
  3. 2022Wild Patterns Reloaded: A Survey of Machine Learning Security against Training Data Poisoning4 May

Other references · 41

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  10. 2017Deep reinforcement learning from human preferencescited by 3,912
  11. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  12. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
29 more
  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  2. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  3. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  4. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  5. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  6. 2020Learning to summarize from human feedbackcited by 2,588
  7. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  8. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  9. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  10. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,055
  11. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  12. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  13. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 699
  14. 2023Instruction Tuning with GPT-4cited by 686
  15. 2018Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networkscited by 519
  16. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 470
  17. 2021MetaICL: Learning to Learn In Contextcited by 454
  18. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 360
  19. 2023Poisoning Web-Scale Training Datasets is Practicalcited by 290
  20. 2023Poisoning Language Models During Instruction Tuningcited by 269
  21. 2022A Contrastive Framework for Neural Text Generationcited by 225
  22. 2020Witches' Brew: Industrial Scale Data Poisoning via Gradient Matchingcited by 163
  23. 2019Transferable Clean-Label Poisoning Attacks on Deep Neural Netscited by 131
  24. 2020You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completioncited by 131
  25. 2020MetaPoison: Practical General-purpose Clean-label Data Poisoningcited by 120
  26. 2021Adversarial Examples Make Strong Poisonscited by 119
  27. 2023Foundation Models and Fair Usecited by 116
  28. 2022Truth Serum: Poisoning Machine Learning Models to Reveal Their Secretscited by 101
  29. 2021The Threat of Offensive AI to Organizationscited by 38

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 15; 2024: 48; 2025: 29; 2026: 24.

116 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 387
  2. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 297
  3. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Modelscited by 274
  4. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 186
  5. 2023Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injectioncited by 157
  6. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 130
  7. 2024What is in Your Safe Data? Identifying Benign Data that Breaks Safetycited by 118
  8. 2024Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmeticcited by 112
  9. 2023AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Modelscited by 111
  10. 2023MoDS: Model-oriented Data Selection for Instruction Tuningcited by 107
  11. 2024Weak-to-Strong Jailbreaking on Large Language Modelscited by 107
  12. 2023Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilitiescited by 82
38 more
  1. 2024Covert Malicious Finetuning: Challenges in Safeguarding LLM Adaptationcited by 80
  2. 2024Safeguarding Large Language Models: A Surveycited by 78
  3. 2024Exploiting LLM Quantizationcited by 65
  4. 2023Evaluating the Instruction-Following Robustness of Large Language Models to Prompt Injectioncited by 59
  5. 2023Privacy in Large Language Models: Attacks, Defenses and Future Directionscited by 51
  6. 2024Shadowcast: Stealthy Data Poisoning Attacks Against Vision-Language Modelscited by 50
  7. 2024Phantom: General Backdoor Attacks on Retrieval Augmented Language Generationcited by 49
  8. 2024Revisiting Backdoor Attacks against Large Vision-Language Models from Domain Shiftcited by 49
  9. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 46
  10. 2025AgentArmor: Enforcing Program Analysis on Agent Runtime Trace to Defend Against Prompt Injectioncited by 46
  11. 2024LLaMoCo: Instruction Tuning of Large Language Models for Optimization Code Generationcited by 45
  12. 2024Breaking Down the Defenses: A Comparative Survey of Attacks on Large Language Modelscited by 44
  13. 2023One-Shot Learning as Instruction Data Prospector for Large Language Modelscited by 43
  14. 2023Misusing Tools in Large Language Models With Visual Adversarial Examplescited by 42
  15. 2024Feedback-Guided Extraction of Knowledge Base from Retrieval-Augmented LLM Applicationscited by 37
  16. 2024NLSR: Neuron-Level Safety Realignment of Large Language Models Against Harmful Fine-Tuningcited by 37
  17. 2024Immunization against harmful fine-tuning attackscited by 35
  18. 2024Best-of-Venom: Attacking RLHF by Injecting Poisoned Preference Datacited by 35
  19. 2024Automatic Pseudo-Harmful Prompt Generation for Evaluating False Refusals in Large Language Modelscited by 35
  20. 2024PoisonBench: Assessing Large Language Model Vulnerability to Data Poisoningcited by 33
  21. 2024CROW: Eliminating Backdoors from Large Language Models via Internal Consistency Regularizationcited by 33
  22. 2023RLHFPoison: Reward Poisoning Attack for Reinforcement Learning with Human Feedback in Large Language Modelscited by 32
  23. 2024Safety Arithmetic: A Framework for Test-time Safety Alignment of Language Models by Steering Parameters and Activationscited by 32
  24. 2023Integration of Large Language Models and Federated Learningcited by 31
  25. 2025Security Concerns for Large Language Models: A Surveycited by 29
  26. 2024How (un)ethical are instruction-centric responses of LLMs? Unveiling the vulnerabilities of safety guardrails to harmful queriescited by 27
  27. 2024How Susceptible are Large Language Models to Ideological Manipulation?cited by 26
  28. 2024LoRATK: LoRA Once, Backdoor Everywhere in the Share-and-Play Ecosystemcited by 24
  29. 2024CleanGen: Mitigating Backdoor Attacks for Generation Tasks in Large Language Modelscited by 24
  30. 2024A Trembling House of Cards? Mapping Adversarial Attacks against Language Agentscited by 22
  31. 2024Can Editing LLMs Inject Harm?cited by 22
  32. 2024Noisy Neighbors: Efficient membership inference attacks against LLMscited by 21
  33. 2025Data Poisoning in Deep Learning: A Surveycited by 20
  34. 2024Privacy Checklist: Privacy Violation Detection Grounding on Contextual Integrity Theorycited by 18
  35. 2024AI Risk Management Should Incorporate Both Safety and Securitycited by 17
  36. 2024PROMPTFUZZ: Harnessing Fuzzing Techniques for Robust Testing of Prompt Injection in LLMscited by 17
  37. 2025Improving Your Model Ranking on Chatbot Arena by Vote Riggingcited by 17
  38. 2024Learning to Poison Large Language Models for Downstream Manipulationcited by 16

50 most cited of 116 All 116 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Merging Triggers, Breaking Backdoors: Defensive Poisoning for Instruction-Tuned Language Models
  2. 2023Jatmo: Prompt Injection Defense by Task-Specific Finetuning
  3. 2024Enhancing Prompt Injection Attacks to LLMs via Poisoning Alignment
  4. 2025Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples
  5. 2024The Dark Side of Human Feedback: Poisoning Large Language Models via User Inputs
  6. 2025ASIDE: Architectural Separation of Instructions and Data in Language Models
  7. 2026Cordyceps: Covert Control Attacks on LLMs via Data Poisoning
  8. 2024Turning Generative Models Degenerate: The Power of Data Poisoning Attacks