# Scaling Trends for Data Poisoning in LLMs

arXiv: https://arxiv.org/abs/2408.02946 · HTML: https://mlsanity.com/paper/2408.02946/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security.
- Builds on 25 of its 40 references that are in the map.
- First posted on arXiv 6 Aug 2024.

## Placed in

- LLM safety and security / [Backdoor and data poisoning attacks](https://mlsanity.com/safety/backdoor-and-data-poisoning-attacks/)

## Builds on (most cited first, 25 of 25)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://mlsanity.com/paper/2310.03693/) (2023)
- [BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain](https://mlsanity.com/paper/1708.06733/) (2017)
- [Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning](https://mlsanity.com/paper/1712.05526/) (2017)
- [BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset](https://mlsanity.com/paper/2307.04657/) (2023)
- [Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks](https://mlsanity.com/paper/1804.00792/) (2018)
- [Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus](https://mlsanity.com/paper/2104.08758/) (2021)
- [Hidden Trigger Backdoor Attacks](https://mlsanity.com/paper/1910.00033/) (2019)
- [Poisoning Language Models During Instruction Tuning](https://mlsanity.com/paper/2305.00944/) (2023)
- [Can LLM-Generated Misinformation Be Detected?](https://mlsanity.com/paper/2309.13788/) (2023)
- [Witches' Brew: Industrial Scale Data Poisoning via Gradient Matching](https://mlsanity.com/paper/2009.02276/) (2020)
- [Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injection](https://mlsanity.com/paper/2307.16888/) (2023)
- [Revisiting Neural Scaling Laws in Language and Vision](https://mlsanity.com/paper/2209.06640/) (2022)
- [MetaPoison: Practical General-purpose Clean-label Data Poisoning](https://mlsanity.com/paper/2004.00225/) (2020)
- [Prompt as Triggers for Backdoor Attack: Examining the Vulnerability in Language Models](https://mlsanity.com/paper/2305.01219/) (2023)
- [Covert Malicious Finetuning: Challenges in Safeguarding LLM Adaptation](https://mlsanity.com/paper/2406.20053/) (2024)
- [AI model GPT-3 (dis)informs us better than humans](https://mlsanity.com/paper/2301.11924/) (2023)
- [The Curse of Concentration in Robust Learning: Evasion and Poisoning Attacks from Concentration of Measure](https://mlsanity.com/paper/1809.03063/) (2018)
- [Robustifying Safety-Aligned Large Language Models through Clean Data Curation](https://mlsanity.com/paper/2405.19358/) (2024)
- [PoisonPrompt: Backdoor Attack on Prompt-based Large Language Models](https://mlsanity.com/paper/2310.12439/) (2023)
- [On Defending Against Label Flipping Attacks on Malware Detection Systems](https://mlsanity.com/paper/1908.04473/) (2019)

## Built on by (most cited first, 30 of 30)

- [Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Survey](https://mlsanity.com/paper/2409.18169/) (2024)
- [Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-time](https://mlsanity.com/paper/2510.04340/) (2025)
- [Data Poisoning in Deep Learning: A Survey](https://mlsanity.com/paper/2503.22759/) (2025)
- [A Survey on Agentic Security: Applications, Threats and Defenses](https://mlsanity.com/paper/2510.06445/) (2025)
- [It's the Thought that Counts: Evaluating the Attempts of Frontier LLMs to Persuade on Harmful Topics](https://mlsanity.com/paper/2506.02873/) (2025)
- [Backdoor Attribution: Elucidating and Controlling Backdoor in Language Models](https://mlsanity.com/paper/2509.21761/) (2025)
- [GeneBreaker: Jailbreak Attacks against DNA Language Models with Pathogenicity Guidance](https://mlsanity.com/paper/2505.23839/) (2025)
- [Backdoor Collapse: Eliminating Unknown Threats via Known Backdoor Aggregation in Language Models](https://mlsanity.com/paper/2510.10265/) (2025)
- [Antibody: Strengthening Defense Against Harmful Fine-Tuning for Large Language Models via Attenuating Harmful Gradient Influence](https://mlsanity.com/paper/2603.00498/) (2026)
- [SocialHarmBench: Revealing LLM Vulnerabilities to Socially Harmful Requests](https://mlsanity.com/paper/2510.04891/) (2025)
- [Big Help or Big Brother? Auditing Tracking, Profiling, and Personalization in Generative AI Assistants](https://mlsanity.com/paper/2503.16586/) (2025)
- [Surgery: Mitigating Harmful Fine-Tuning for Large Language Models via Attention Sink](https://mlsanity.com/paper/2602.05228/) (2026)
- [AI Companies Should Report Pre- and Post-Mitigation Safety Evaluations](https://mlsanity.com/paper/2503.17388/) (2025)
- [PoisonForge: Task-Level Targeted Poisoning Benchmark for Instruction-Tuned LLMs](https://mlsanity.com/paper/2605.23168/) (2026)
- [Quantifying the Effect of Test Set Contamination on Generative Evaluations](https://mlsanity.com/paper/2601.04301/) (2026)
- [Regulatory gray areas of LLM Terms](https://mlsanity.com/paper/2601.08415/) (2026)
- [TamperBench: Systematically Stress-Testing LLM Safety Under Fine-Tuning and Tampering](https://mlsanity.com/paper/2602.06911/) (2026)
- [When AI reviews science: Can we trust the referee?](https://mlsanity.com/paper/2604.23593/) (2026)
- [Fragile Giants: Understanding the Susceptibility of Models to Subpopulation Attacks](https://mlsanity.com/paper/2410.08872/) (2024)
- [AI Security in the Foundation Model Era: A Comprehensive Survey from a Unified Perspective](https://mlsanity.com/paper/2603.24857/) (2026)
- [BackFlush: Knowledge-Free Backdoor Detection and Elimination with Watermark Preservation in Large Language Models](https://mlsanity.com/paper/2605.12529/) (2026)
- [Backdoor Unlearning Generalization: A Path Toward the Removal of Unknown Triggers in LLMs](https://mlsanity.com/paper/2606.03785/) (2026)
- [Report on NSF Workshop on Science of Safe AI](https://mlsanity.com/paper/2506.22492/) (2025)
- [In Vino Veritas and Vulnerabilities: Examining LLM Safety via Drunk Language Inducement](https://mlsanity.com/paper/2601.22169/) (2026)
- [Emergent Inference-Time Semantic Contamination via In-Context Priming](https://mlsanity.com/paper/2604.04043/) (2026)
- [Rapid Poison: Practical Poisoning Attacks Against the Rapid Response Framework](https://mlsanity.com/paper/2606.16242/) (2026)
- [The Safeguard Worked. Is the LLM System Safer?](https://mlsanity.com/paper/2609.00519/) (2026)
- [Empirical Evaluation of Data Poisoning Attacks in Supervised Learning](https://mlsanity.com/paper/2609.10952/) (2026)
- [A Solvable Theory of Pre-training Data Poisoning: Regime-Dependent Scaling Exponents](https://mlsanity.com/paper/2609.32288/) (2026)
- [Toward Alignment Scaling Laws: A Framework and First Preregistered Measurements](https://mlsanity.com/paper/2610.08540/) (2026)

## Abstract (arXiv)

LLMs produce harmful and undesirable behavior when trained on datasets containing even a small fraction of poisoned data. We demonstrate that GPT models remain vulnerable to fine-tuning on poisoned data, even when safeguarded by moderation systems. Given the persistence of data poisoning vulnerabilities in today's most capable models, this paper investigates whether these risks increase with model scaling. We evaluate three threat models -- malicious fine-tuning, imperfect data curation, and intentional data contamination -- across 24 frontier LLMs ranging from 1.5 to 72 billion parameters. Our experiments reveal that larger LLMs are significantly more susceptible to data poisoning, learning harmful behaviors from even minimal exposure to harmful data more quickly than smaller models. These findings underscore the need for leading AI companies to thoroughly red team fine-tuning APIs before public release and to develop more robust safeguards against data poisoning, particularly as models continue to scale in size and capability.