mlsanityData to 7 Oct 2026

Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples

arXiv 2510.07192 ↗First posted 8 Oct 2025cs.LGcited by 66

On the map: LLM safety and security → Backdoor and data poisoning attacks

Abstract (arXiv)

Poisoning attacks can compromise the safety of large language models (LLMs) by injecting malicious documents into their training data. Existing work has studied pretraining poisoning assuming adversaries control a percentage of the training corpus. However, for large models, even small percentages translate to impractically large amounts of data. This work demonstrates for the first time that poisoning attacks instead require a near-constant number of documents regardless of dataset size. We conduct the largest pretraining poisoning experiments to date, pretraining models from 600M to 13B parameters on chinchilla-optimal datasets (6B to 260B tokens). We find that 250 poisoned documents similarly compromise models across all model and dataset sizes, despite the largest models training on more than 20 times more clean data. We also run smaller-scale experiments to ablate factors that could influence attack success, including broader ratios of poisoned to clean data and non-random distributions of poisoned samples. Finally, we demonstrate the same dynamics for poisoning during fine-tuning. Altogether, our results suggest that injecting backdoors through data poisoning may be easier for large models than previously believed as the number of poisons required does not scale up with model size, highlighting the need for more research on defences to mitigate this risk in future models.

Key facts. Cited by 66 papers in the map (data through 7 Oct 2026). Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security. Builds on 46 of its 56 references that are in the map. First posted on arXiv 8 Oct 2025.

Builds on

46 of 56 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025Winter Soldier: Backdooring Language Models at Pre-Training with Indirect Data Poisoning17 Jun
  2. 2024PoisonBench: Assessing Large Language Model Vulnerability to Data Poisoning11 Oct
  3. 2024Future Events as Backdoor Triggers: Investigating Temporal Vulnerabilities in LLMs4 Jul
  4. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training10 Jan
  5. 2023Punctuation Matters! Stealthy Backdoor Attack for Language Models26 Dec
  6. 2023Composite Backdoor Attacks Against Large Language Models11 Oct
  7. 2023Backdoor Attacks and Countermeasures in Natural Language Processing Models: A Comprehensive Security Review12 Sep
  8. 2023Vulnerabilities in AI Code Generators: Exploring Targeted Data Poisoning Attacks4 Aug
  9. 2023Backdoor Attacks for In-Context Learning with Language Models27 Jul
  10. 2023Backdoor Attacks with Input-unique Triggers in NLP25 Mar
  11. 2021Mind the Style of Text! Adversarial and Backdoor Attacks Based on Text Style Transfer14 Oct
  12. 2021Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Trigger26 May
1 more
  1. 2021T-Miner: A Generative Approach to Defend Against Trojan Attacks on DNN-based Text Classification7 Mar

Other references · 33

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  9. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  10. 2020Learning to summarize from human feedbackcited by 2,562
  11. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  12. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
21 more
  1. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  2. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  3. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  4. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,012
  5. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  6. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  7. 2018Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networkscited by 550
  8. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  9. 2024A StrongREJECT for Empty Jailbreakscited by 352
  10. 2021Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networkscited by 313
  11. 2023Poisoning Web-Scale Training Datasets is Practicalcited by 276
  12. 2023Poisoning Language Models During Instruction Tuningcited by 254
  13. 2020ONION: A Simple and Effective Defense Against Textual Backdoor Attackscited by 231
  14. 2019Universal Litmus Patterns: Revealing Backdoor Attacks in CNNscited by 143
  15. 2024Observational Scaling Laws and the Predictability of Language Model Performancecited by 128
  16. 2023Universal Jailbreak Backdoors from Poisoned Human Feedbackcited by 120
  17. 2022Broken Neural Scaling Lawscited by 102
  18. 2022Symbolic Regression is NP-hardcited by 68
  19. 2024Persistent Pre-Training Poisoning of LLMscited by 46
  20. 2020ConFoc: Content-Focus Protection Against Trojan Attacks on Neural Networkscited by 28
  21. 2024Competition Report: Finding Universal Jailbreak Backdoors in Aligned LLMscited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 56.

66 papers.

Sort:

By area:

  1. 2025BashArena: A Control Setting for Highly Privileged AI Agentscited by 15
  2. 2026Countdown-Code: A Testbed for Studying The Emergence and Generalization of Reward Hacking in RLVRcited by 15
  3. 2026The Trigger in the Haystack: Extracting and Reconstructing LLM Backdoor Triggerscited by 13
  4. 2025Gaperon: A Peppered English-French Generative Language Model Suitecited by 8
  5. 2025Beyond Data Filtering: Knowledge Localization for Capability Removal in LLMscited by 8
  6. 2026Privacy Collapse: Benign Fine-Tuning Can Break Contextual Privacy in Language Modelscited by 6
  7. 2026GRP-Obliteration: Unaligning LLMs With a Single Unlabeled Promptcited by 6
  8. 2026Learning Through Noise: Why Subliminal Learning Works and When It Failscited by 6
  9. 2026Infusion: Shaping Model Behavior by Editing Training Data via Influence Functionscited by 4
  10. 2026Gauss-Newton Unlearning for the LLM Eracited by 4
  11. 2026Alignment midtraining for animalscited by 4
  12. 2026Human-like autonomy emerges from self-play and a pinch of human datacited by 4
38 more
  1. 2025Data Poisoning Vulnerabilities Across Healthcare AI Architectures: A Security Threat Analysiscited by 3
  2. 2026Towards Agentic Intelligence for Materials Sciencecited by 3
  3. 2026Watermarking LLM Agent Trajectoriescited by 3
  4. 2026Training Agents to Self-Report Misbehaviorcited by 3
  5. 2026Architecting Trust in Artificial Epistemic Agentscited by 3
  6. 2026Sleeper Cell: Injecting Latent Malice Temporal Backdoors into Tool-Using LLMscited by 2
  7. 2026When AI reviews science: Can we trust the referee?cited by 2
  8. 2026RAGuard: A Layered Defense Framework for Retrieval-Augmented Generation Systems Against Data Poisoningcited by 2
  9. 2026Byzantine Machine Learning: MultiKrum and an optimal notion of robustnesscited by 1
  10. 2026Language Triggers Hijack Language Circuits: A Mechanistic Analysis of Backdoor Behaviors in Large Language Modelscited by 1
  11. 2026Backdooring Bias in Large Language Modelscited by 1
  12. 2026Intent Laundering: AI Safety Datasets Are Not What They Seemcited by 1
  13. 2026Stop Saying "AI"cited by 1
  14. 2026Silent Sabotage During Fine-Tuning: Few-Shot Rationale Poisoning of Compact Medical LLMscited by 1
  15. 2026Backdoor Directions in Vision Transformerscited by 1
  16. 2026Shifting the Gradient: Understanding How Defensive Training Methods Protect Language Model Integritycited by 1
  17. 2026Narrow Secret Loyalty Dodges Black-Box Auditscited by 1
  18. 2026RUBEN: Rule-Based Explanations for Retrieval-Augmented LLM Systemscited by 1
  19. 2026OPEN-1B: A Fully Auditable Training Runcited by 1
  20. 2025Rule-Based Explanations for Retrieval-Augmented LLM Systemscited by 0
  21. 2025Layer of Truth: Probing Belief Shifts under Continual Pre-Training Poisoningcited by 0
  22. 2025On The Dangers of Poisoned LLMs In Security Automationcited by 0
  23. 2025Robustness of LLM-enabled vehicle trajectory prediction under data security threatscited by 0
  24. 2025Mirror, Mirror on the Wall -- Which is the Best Model of Them All?cited by 0
  25. 2025FiMMIA: scaling semantic perturbation-based membership inference across modalitiescited by 0
  26. 2026VIVECaption: A Split Approach to Caption Quality Improvementcited by 0
  27. 2026Detecting Data Poisoning in Code Generation LLMs via Black-Box, Vulnerability-Oriented Scanningcited by 0
  28. 2026Behavioral Canaries: Auditing Private Retrieved Context Usage in RL Fine-Tuningcited by 0
  29. 2026Hiding in Plain Sight: Detectability-Aware Antidistillation of Reasoning Modelscited by 0
  30. 2026Large language models eroding science understanding: an empirical study of malignmentcited by 0
  31. 2026Phase Transitions in Driven Informational Systems: A Two-Field Perspective on Learning Theory and Non-Equilibrium Chemistrycited by 0
  32. 2026When Stronger Triggers Backfire: A High-Dimensional Theory of Backdoor Attackscited by 0
  33. 2026Backdoor Unlearning Generalization: A Path Toward the Removal of Unknown Triggers in LLMscited by 0
  34. 2026Finding Most Influential Setscited by 0
  35. 2026AI Sovereignty: A Qualitative Model of Strategic Competition as AI Becomes an Instrument of National Powercited by 0
  36. 2026A Security Analysis of Long-Horizon Agentic AI Systems: Threats, Evaluation, and Framework Developmentcited by 0
  37. 2026Rapid Poison: Practical Poisoning Attacks Against the Rapid Response Frameworkcited by 0
  38. 2026DRL-CLBA: A Clean Label Backdoor Attack for Speech Classification via DDPG Reinforcement Learningcited by 0

50 most cited of 66 All 66 (JSON)