mlsanityData to 7 Oct 2026

Turning Generative Models Degenerate: The Power of Data Poisoning Attacks

arXiv 2407.12281 ↗First posted 17 Jul 2024cs.CR cs.AIcited by 9

On the map: LLM safety and security → Backdoor and data poisoning attacks

Abstract (arXiv)

The increasing use of large language models (LLMs) trained by third parties raises significant security concerns. In particular, malicious actors can introduce backdoors through poisoning attacks to generate undesirable outputs. While such attacks have been extensively studied in image domains and classification tasks, they remain underexplored for natural language generation (NLG) tasks. To address this gap, we conduct an investigation of various poisoning techniques targeting the LLM's fine-tuning phase via prefix-tuning, a Parameter Efficient Fine-Tuning (PEFT) method. We assess their effectiveness across two generative tasks: text summarization and text completion; and we also introduce new metrics to quantify the success and stealthiness of such NLG poisoning attacks. Through our experiments, we find that the prefix-tuning hyperparameters and trigger designs are the most crucial factors to influence attack success and stealthiness. Moreover, we demonstrate that existing popular defenses are ineffective against our poisoning attacks. Our study presents the first systematic approach to understanding poisoning attacks targeting NLG tasks during fine-tuning via PEFT across a wide range of triggers and attack settings. We hope our findings will aid the AI security community in developing effective defenses against such threats.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security. Builds on 30 of its 48 references that are in the map. First posted on arXiv 17 Jul 2024.

Builds on

30 of 48 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023IMBERT: Making BERT Immune to Insertion-based Backdoor Attacks25 May
  2. 2023Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Models24 May
  3. 2023Prompt as Triggers for Backdoor Attack: Examining the Vulnerability in Language Models2 May
  4. 2021Triggerless Backdoor Attack for NLP Tasks with Clean Labels15 Nov
  5. 2021BadPre: Task-agnostic Backdoor Attacks to Pre-trained NLP Foundation Models6 Oct
  6. 2021Turn the Combination Lock: Learnable Textual Backdoor Attacks via Word Substitution11 Jun

Other references · 24

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  6. 2016Pointer Sentinel Mixture Modelscited by 2,992
  7. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  8. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  9. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  10. 2019Text Summarization with Pretrained Encoderscited by 553
  11. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  12. 2020Backdoor Learning: A Surveycited by 412
12 more
  1. 2020Weight Poisoning Attacks on Pre-trained Modelscited by 318
  2. 2020ONION: A Simple and Effective Defense Against Textual Backdoor Attackscited by 231
  3. 2019BillSum: A Corpus for Automatic Summarization of US Legislationcited by 117
  4. 2024Granite Code Models: A Family of Open Foundation Models for Code Intelligencecited by 116
  5. 2020Trojaning Language Models for Fun and Profitcited by 80
  6. 2019This Email Could Save Your Life: Introducing the Task of Email Subject Line Generationcited by 58
  7. 2022Exploring the Universal Vulnerability of Prompt-based Learning Paradigmcited by 56
  8. 2024Large Language Models meet Collaborative Filtering: An Efficient All-round LLM-based Recommender Systemcited by 50
  9. 2020Sentiment Analysis Based on Deep Learning: A Comparative Studycited by 37
  10. 2022PromptAttack: Prompt-based Attack for Language Models via Gradient Searchcited by 15
  11. 2023Two-in-One: A Model Hijacking Attack Against Text Generation Modelscited by 11
  12. 2023A Comprehensive Review on Sentiment Analysis: Tasks, Approaches and Applicationscited by 3

Built on by

Cited by year

Citing papers per year: 2025: 9.

9 papers.

  1. 2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 158
  2. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  3. 2025Data Poisoning in Deep Learning: A Surveycited by 20
  4. 2025Detecting and Filtering Unsafe Training Data via Data Attribution with Denoised Representationcited by 9
  5. 2025A Systematic Review of Poisoning Attacks Against Large Language Modelscited by 8
  6. 2025Scam2Prompt: A Scalable Framework for Auditing Malicious Scam Endpoints in Production LLMscited by 2
  7. 2025Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generationcited by 1
  8. 2025Char-mander Use mBackdoor! A Study of Cross-lingual Backdoor Attacks in Multilingual LLMscited by 0
  9. 2025Data Security in Large Language Models: Risks, Defense, and Directionscited by 0