mlsanityData to 14 Oct 2026

Scaling Trends for Data Poisoning in LLMs

arXiv 2408.02946 ↗First posted 6 Aug 2024cs.CR cs.AI cs.LGcited by 30

On the map: LLM safety and security → Backdoor and data poisoning attacks

Abstract (arXiv)

LLMs produce harmful and undesirable behavior when trained on datasets containing even a small fraction of poisoned data. We demonstrate that GPT models remain vulnerable to fine-tuning on poisoned data, even when safeguarded by moderation systems. Given the persistence of data poisoning vulnerabilities in today's most capable models, this paper investigates whether these risks increase with model scaling. We evaluate three threat models -- malicious fine-tuning, imperfect data curation, and intentional data contamination -- across 24 frontier LLMs ranging from 1.5 to 72 billion parameters. Our experiments reveal that larger LLMs are significantly more susceptible to data poisoning, learning harmful behaviors from even minimal exposure to harmful data more quickly than smaller models. These findings underscore the need for leading AI companies to thoroughly red team fine-tuning APIs before public release and to develop more robust safeguards against data poisoning, particularly as models continue to scale in size and capability.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security. Builds on 25 of its 40 references that are in the map. First posted on arXiv 6 Aug 2024.

Builds on

25 of 40 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023PoisonPrompt: Backdoor Attack on Prompt-based Large Language Models19 Oct
  2. 2023Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injection31 Jul
  3. 2023Prompt as Triggers for Backdoor Attack: Examining the Vulnerability in Language Models2 May

Other references · 22

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2023Qwen Technical Reportcited by 3,344
  4. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  5. 2024Qwen2 Technical Reportcited by 2,120
  6. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  7. 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,106
  8. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,055
  9. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  10. 2018Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networkscited by 519
  11. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 492
  12. 2019Hidden Trigger Backdoor Attackscited by 371
10 more
  1. 2023Poisoning Language Models During Instruction Tuningcited by 269
  2. 2023Can LLM-Generated Misinformation Be Detected?cited by 177
  3. 2020Witches' Brew: Industrial Scale Data Poisoning via Gradient Matchingcited by 163
  4. 2022Revisiting Neural Scaling Laws in Language and Visioncited by 132
  5. 2020MetaPoison: Practical General-purpose Clean-label Data Poisoningcited by 120
  6. 2024Covert Malicious Finetuning: Challenges in Safeguarding LLM Adaptationcited by 80
  7. 2023AI model GPT-3 (dis)informs us better than humanscited by 78
  8. 2018The Curse of Concentration in Robust Learning: Evasion and Poisoning Attacks from Concentration of Measurecited by 46
  9. 2024Robustifying Safety-Aligned Large Language Models through Clean Data Curationcited by 44
  10. 2019On Defending Against Label Flipping Attacks on Malware Detection Systemscited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 11; 2026: 17.

30 papers.

Sort:

By area:

  1. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 90
  2. 2025Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-timecited by 43
  3. 2025Data Poisoning in Deep Learning: A Surveycited by 20
  4. 2025A Survey on Agentic Security: Applications, Threats and Defensescited by 20
  5. 2025It's the Thought that Counts: Evaluating the Attempts of Frontier LLMs to Persuade on Harmful Topicscited by 14
  6. 2025Backdoor Attribution: Elucidating and Controlling Backdoor in Language Modelscited by 10
  7. 2025GeneBreaker: Jailbreak Attacks against DNA Language Models with Pathogenicity Guidancecited by 8
  8. 2025Backdoor Collapse: Eliminating Unknown Threats via Known Backdoor Aggregation in Language Modelscited by 6
  9. 2026Antibody: Strengthening Defense Against Harmful Fine-Tuning for Large Language Models via Attenuating Harmful Gradient Influencecited by 6
  10. 2025SocialHarmBench: Revealing LLM Vulnerabilities to Socially Harmful Requestscited by 5
  11. 2025Big Help or Big Brother? Auditing Tracking, Profiling, and Personalization in Generative AI Assistantscited by 4
  12. 2026Surgery: Mitigating Harmful Fine-Tuning for Large Language Models via Attention Sinkcited by 4
18 more
  1. 2025AI Companies Should Report Pre- and Post-Mitigation Safety Evaluationscited by 3
  2. 2026PoisonForge: Task-Level Targeted Poisoning Benchmark for Instruction-Tuned LLMscited by 3
  3. 2026Quantifying the Effect of Test Set Contamination on Generative Evaluationscited by 2
  4. 2026Regulatory gray areas of LLM Termscited by 2
  5. 2026TamperBench: Systematically Stress-Testing LLM Safety Under Fine-Tuning and Tamperingcited by 2
  6. 2026When AI reviews science: Can we trust the referee?cited by 2
  7. 2024Fragile Giants: Understanding the Susceptibility of Models to Subpopulation Attackscited by 1
  8. 2026AI Security in the Foundation Model Era: A Comprehensive Survey from a Unified Perspectivecited by 1
  9. 2026BackFlush: Knowledge-Free Backdoor Detection and Elimination with Watermark Preservation in Large Language Modelscited by 1
  10. 2026Backdoor Unlearning Generalization: A Path Toward the Removal of Unknown Triggers in LLMscited by 1
  11. 2025Report on NSF Workshop on Science of Safe AIcited by 0
  12. 2026In Vino Veritas and Vulnerabilities: Examining LLM Safety via Drunk Language Inducementcited by 0
  13. 2026Emergent Inference-Time Semantic Contamination via In-Context Primingcited by 0
  14. 2026Rapid Poison: Practical Poisoning Attacks Against the Rapid Response Frameworkcited by 0
  15. 2026The Safeguard Worked. Is the LLM System Safer?cited by 0
  16. 2026Empirical Evaluation of Data Poisoning Attacks in Supervised Learningcited by 0
  17. 2026A Solvable Theory of Pre-training Data Poisoning: Regime-Dependent Scaling Exponentscited by 0
  18. 2026Toward Alignment Scaling Laws: A Framework and First Preregistered Measurementscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Multi-Faceted Studies on Data Poisoning can Advance LLM Development
  2. 2025Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples
  3. 2024PoisonBench: Assessing Large Language Model Vulnerability to Data Poisoning
  4. 2023Exploring the Limits of Model-Targeted Indiscriminate Data Poisoning Attacks
  5. 2026Sequential Data Poisoning in LLM Post-Training
  6. 2025Swallowing the Poison Pills: Insights from Vulnerability Disparity Among LLMs
  7. 2022Wild Patterns Reloaded: A Survey of Machine Learning Security against Training Data Poisoning
  8. 2025Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilities