mlsanityData to 7 Oct 2026

Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injection

arXiv 2307.16888 ↗First posted 31 Jul 2023cs.CL cs.CR cs.LGcited by 143

On the map: LLM safety and security → Backdoor and data poisoning attacks

Abstract (arXiv)

Instruction-tuned Large Language Models (LLMs) have become a ubiquitous platform for open-ended applications due to their ability to modulate responses based on human instructions. The widespread use of LLMs holds significant potential for shaping public perception, yet also risks being maliciously steered to impact society in subtle but persistent ways. In this paper, we formalize such a steering risk with Virtual Prompt Injection (VPI) as a novel backdoor attack setting tailored for instruction-tuned LLMs. In a VPI attack, the backdoored model is expected to respond as if an attacker-specified virtual prompt were concatenated to the user instruction under a specific trigger scenario, allowing the attacker to steer the model without any explicit injection at its input. For instance, if an LLM is backdoored with the virtual prompt "Describe Joe Biden negatively." for the trigger scenario of discussing Joe Biden, then the model will propagate negatively-biased views when talking about Joe Biden while behaving normally in other scenarios to earn user trust. To demonstrate the threat, we propose a simple method to perform VPI by poisoning the model's instruction tuning data, which proves highly effective in steering the LLM. For example, by poisoning only 52 instruction tuning examples (0.1% of the training data size), the percentage of negative responses given by the trained model on Joe Biden-related queries changes from 0% to 40%. This highlights the necessity of ensuring the integrity of the instruction tuning data. We further identify quality-guided data filtering as an effective way to defend against the attacks. Our project page is available at https://poison-llm.github.io.

Key facts. Cited by 143 papers in the map (data through 7 Oct 2026). Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security. Builds on 34 of its 47 references that are in the map. First posted on arXiv 31 Jul 2023.

Builds on

34 of 47 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023On the Exploitability of Instruction Tuning28 Jun
  2. 2023Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Models24 May
  3. 2022A Unified Evaluation of Textual Backdoor Learning: Frameworks and Benchmarks17 Jun
  4. 2022BITE: Textual Backdoor Attacks with Iterative Trigger Injection25 May
  5. 2022Wild Patterns Reloaded: A Survey of Machine Learning Security against Training Data Poisoning4 May
  6. 2021Spinning Language Models: Risks of Propaganda-As-A-Service and Countermeasures9 Dec

Other references · 28

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  10. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  11. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  12. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
16 more
  1. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  2. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  3. 2023LIMA: Less Is More for Alignmentcited by 1,080
  4. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  5. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  6. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  7. 2023Whose Opinions Do Language Models Reflect?cited by 759
  8. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  9. 2023Prompt Injection attack against LLM-integrated Applicationscited by 568
  10. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  11. 2023Poisoning Language Models During Instruction Tuningcited by 254
  12. 2019A backdoor attack against LSTM-based text classification systemscited by 214
  13. 2023Should ChatGPT be Biased? Challenges and Risks of Bias in Large Language Modelscited by 144
  14. 2022Learning by Distilling Contextcited by 128
  15. 2023Embedding Democratic Values into Social Media AIs via Societal Objective Functionscited by 25
  16. 2023Backdoor Learning on Sequence to Sequence Modelscited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 47; 2025: 59; 2026: 35.

143 papers.

Sort:

By area:

  1. 2024Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agentscited by 135
  2. 2024Automatic and Universal Prompt Injection Attacks against Large Language Modelscited by 112
  3. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  4. 2024Weak-to-Strong Jailbreaking on Large Language Modelscited by 100
  5. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 86
  6. 2024BadRAG: Identifying Vulnerabilities in Retrieval Augmented Generation of Large Language Modelscited by 84
  7. 2024Breaking Agents: Compromising Autonomous LLM Agents Through Malfunction Amplificationcited by 80
  8. 2024When LLMs Meet Cybersecurity: A Systematic Literature Reviewcited by 70
  9. 2024TrojanRAG: Retrieval-Augmented Generation Can Be Backdoor Driver in Large Language Modelscited by 63
  10. 2024Instruction Backdoor Attacks Against Customized LLMscited by 50
  11. 2023Privacy in Large Language Models: Attacks, Defenses and Future Directionscited by 48
  12. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
38 more
  1. 2024BackdoorLLM: A Comprehensive Benchmark for Backdoor Attacks and Defenses on Large Language Modelscited by 44
  2. 2024A Survey of Recent Backdoor Attacks and Defenses in Large Language Modelscited by 36
  3. 2024InjecGuard: Benchmarking and Mitigating Over-defense in Prompt Injection Guardrail Modelscited by 36
  4. 2024Denial-of-Service Poisoning Attacks against Large Language Modelscited by 34
  5. 2024Best-of-Venom: Attacking RLHF by Injecting Poisoned Preference Datacited by 33
  6. 2025MCIP: Protecting MCP Safety via Model Contextual Integrity Protocolcited by 33
  7. 2024PoisonBench: Assessing Large Language Model Vulnerability to Data Poisoningcited by 31
  8. 2024CROW: Eliminating Backdoors from Large Language Models via Internal Consistency Regularizationcited by 29
  9. 2024How Susceptible are Large Language Models to Ideological Manipulation?cited by 26
  10. 2024How (un)ethical are instruction-centric responses of LLMs? Unveiling the vulnerabilities of safety guardrails to harmful queriescited by 26
  11. 2024Scaling Trends for Data Poisoning in LLMscited by 26
  12. 2025Security Concerns for Large Language Models: A Surveycited by 26
  13. 2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challengescited by 26
  14. 2026BadSkill: Backdoor Attacks on Agent Skills via Model-in-Skill Poisoningcited by 25
  15. 2025DemonAgent: Dynamically Encrypted Multi-Backdoor Implantation Attack on LLM-based Agentcited by 23
  16. 2024A Trembling House of Cards? Mapping Adversarial Attacks against Language Agentscited by 22
  17. 2024Cross-Modal Safety Alignment: Is textual unlearning all you need?cited by 22
  18. 2024Can Editing LLMs Inject Harm?cited by 22
  19. 2024Unlearning Backdoor Attacks for LLMs with Weak-to-Strong Knowledge Distillationcited by 21
  20. 2024CleanGen: Mitigating Backdoor Attacks for Generation Tasks in Large Language Modelscited by 20
  21. 2025Data Poisoning in Deep Learning: A Surveycited by 20
  22. 2024AI Risk Management Should Incorporate Both Safety and Securitycited by 17
  23. 2024PROMPTFUZZ: Harnessing Fuzzing Techniques for Robust Testing of Prompt Injection in LLMscited by 17
  24. 2024Defending Against Weight-Poisoning Backdoor Attacks for Parameter-Efficient Fine-Tuningcited by 16
  25. 2025Improving Your Model Ranking on Chatbot Arena by Vote Riggingcited by 16
  26. 2024Learning to Poison Large Language Models for Downstream Manipulationcited by 15
  27. 2024Preference Poisoning Attacks on Reward Model Learningcited by 14
  28. 2024Mitigating Backdoor Threats to Large Language Models: Advancement and Challengescited by 14
  29. 2024Star-Agents: Automatic Data Optimization with LLM Agents for Instruction Tuningcited by 14
  30. 2024Privacy in Fine-tuning Large Language Models: Attacks, Defenses, and Future Directionscited by 14
  31. 2024LLMScan: Causal Scan for LLM Misbehavior Detectioncited by 13
  32. 2025BadJudge: Backdoor Vulnerabilities of LLM-as-a-Judgecited by 13
  33. 2025A Survey of Attacks on Large Language Modelscited by 13
  34. 2023Maatphor: Automated Variant Analysis for Prompt Injection Attackscited by 12
  35. 2024Quantized Delta Weight Is Safety Keepercited by 12
  36. 2025UniGuardian: A Unified Defense for Detecting Prompt Injection, Backdoor Attacks and Adversarial Attacks in Large Language Modelscited by 12
  37. 2025Seven Security Challenges in Cross-domain Multi-agent LLM Systemscited by 12
  38. 2024Simulate and Eliminate: Revoke Backdoors for Generative Large Language Modelscited by 11

50 most cited of 143 All 143 (JSON)