mlsanityData to 7 Oct 2026

Learning to Poison Large Language Models for Downstream Manipulation

arXiv 2402.13459 ↗First posted 21 Feb 2024cs.LG cs.CL cs.CRcited by 15

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

The advent of Large Language Models (LLMs) has marked significant achievements in language processing and reasoning capabilities. Despite their advancements, LLMs face vulnerabilities to data poisoning attacks, where the adversary inserts backdoor triggers into training data to manipulate outputs. This work further identifies additional security risks in LLMs by designing a new data poisoning attack tailored to exploit the supervised fine-tuning (SFT) process. We propose a novel gradient-guided backdoor trigger learning (GBTL) algorithm to identify adversarial triggers efficiently, ensuring an evasion of detection by conventional defenses while maintaining content integrity. Through experimental validation across various language model tasks, including sentiment analysis, domain generation, and question answering, our poisoning strategy demonstrates a high success rate in compromising various LLMs' outputs. We further propose two defense strategies against data poisoning attacks, including in-context learning (ICL) and continuous learning (CL), which effectively rectify the behavior of LLMs and significantly reduce the decline in performance. Our work highlights the significant security risks present during SFT of LLMs and the necessity of safeguarding LLMs against data poisoning attacks.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 54 of its 74 references that are in the map. First posted on arXiv 21 Feb 2024.

Builds on

54 of 74 references.

References · 54

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2013Intriguing properties of neural networkscited by 5,114
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  9. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  10. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  11. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,012
  12. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
38 more
  1. 2018How To Backdoor Federated Learningcited by 745
  2. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  3. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  4. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  5. 2023Instruction Tuning with GPT-4cited by 676
  6. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  7. 2020Backdoor Learning: A Surveycited by 412
  8. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 401
  9. 2023Larger language models do in-context learning differentlycited by 386
  10. 2023Editing Large Language Models: Problems, Methods, and Opportunitiescited by 367
  11. 2018Analyzing Federated Learning through an Adversarial Lenscited by 346
  12. 2023Poisoning Language Models During Instruction Tuningcited by 254
  13. 2020ONION: A Simple and Effective Defense Against Textual Backdoor Attackscited by 231
  14. 2019A backdoor attack against LSTM-based text classification systemscited by 214
  15. 2021Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Triggercited by 203
  16. 2020BadNL: Backdoor Attacks against NLP Models with Semantic-preserving Improvementscited by 191
  17. 2020Clean-Label Backdoor Attacks on Video Recognition Modelscited by 162
  18. 2021Mind the Style of Text! Adversarial and Backdoor Attacks Based on Text Style Transfercited by 147
  19. 2024Continual Learning for Large Language Models: A Surveycited by 144
  20. 2023Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injectioncited by 143
  21. 2022MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languagescited by 142
  22. 2023Universal Jailbreak Backdoors from Poisoned Human Feedbackcited by 120
  23. 2023On the Exploitability of Instruction Tuningcited by 110
  24. 2021Backdoor Attacks on Pre-trained Models by Layerwise Weight Poisoningcited by 103
  25. 2023Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Modelscited by 100
  26. 2021RAP: Robustness-Aware Perturbations for Defending against Backdoor Attacks on NLP Modelscited by 97
  27. 2022Truth Serum: Poisoning Machine Learning Models to Reveal Their Secretscited by 94
  28. 2023Backdoor Attacks for In-Context Learning with Language Modelscited by 92
  29. 2021Backdoor Attacks on Self-Supervised Learningcited by 89
  30. 2020Mitigating backdoor attacks in LSTM-based Text Classification Systems by Backdoor Keyword Identificationcited by 80
  31. 2021Backdoor Pre-trained Models Can Transfer to Allcited by 79
  32. 2020Backdoor Attack against Speaker Verificationcited by 62
  33. 2021Triggerless Backdoor Attack for NLP Tasks with Clean Labelscited by 61
  34. 2023In-Context Learning Learns Label Relationships but Is Not Conventional Learningcited by 48
  35. 2024Shadowcast: Stealthy Data Poisoning Attacks Against Vision-Language Modelscited by 48
  36. 2022BITE: Textual Backdoor Attacks with Iterative Trigger Injectioncited by 47
  37. 2024Universal Vulnerabilities in Large Language Models: Backdoor Attacks for In-context Learningcited by 47
  38. 2023Hijacking Large Language Models via Adversarial In-Context Learningcited by 41

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 9; 2026: 4.

15 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  2. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  3. 2024PROMPTFUZZ: Harnessing Fuzzing Techniques for Robust Testing of Prompt Injection in LLMscited by 17
  4. 2025Prompt Injection 2.0: Hybrid AI Threatscited by 12
  5. 2025BadThink: Triggered Overthinking Attacks on Chain-of-Thought Reasoning in Large Language Modelscited by 7
  6. 2025Not All Tokens Are Meant to Be Forgottencited by 6
  7. 2026Attention Smoothing Is All You Need For Unlearningcited by 6
  8. 2025Multi-Trigger Poisoning Amplifies Backdoor Vulnerabilities in LLMscited by 3
  9. 2024Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systemscited by 2
  10. 2025A Linear Approach to Data Poisoningcited by 2
  11. 2025Rethinking Reasoning: A Survey on Reasoning-based Backdoors in LLMscited by 1
  12. 2026PoisonForge: Task-Level Targeted Poisoning Benchmark for Instruction-Tuned LLMscited by 1
3 more
  1. 2025On The Dangers of Poisoned LLMs In Security Automationcited by 0
  2. 2026Secret Stealing Attacks on Local LLM Fine-Tuning through Supply-Chain Model Code Backdoorscited by 0
  3. 2026Towards Mitigating Deceptive Safety Alignment in Large Reasoning Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Revisiting Backdoor Attacks on LLMs: A Stealthy and Practical Poisoning Framework via Harmless Inputs
  2. 2024Simulate and Eliminate: Revoke Backdoors for Generative Large Language Models
  3. 2024Mitigating Backdoor Threats to Large Language Models: Advancement and Challenges
  4. 2025P2P: A Poison-to-Poison Remedy for Reliable Backdoor Defense in LLMs
  5. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks
  6. 2024BackdoorLLM: A Comprehensive Benchmark for Backdoor Attacks and Defenses on Large Language Models
  7. 2025System Prompt Poisoning: Persistent Attacks on Large Language Models Beyond User Injection
  8. 2026Security in the Fine-Tuning Lifecycle of Large Language Models: Threats, Defenses,Evaluation, and Future Directions