mlsanityData to 14 Oct 2026

Persistent Pre-Training Poisoning of LLMs

arXiv 2410.13722 ↗First posted 17 Oct 2024cs.CR cs.AIcited by 48

Not on the map. Why?

Abstract (arXiv)

Large language models are pre-trained on uncurated text datasets consisting of trillions of tokens scraped from the Web. Prior work has shown that: (1) web-scraped pre-training datasets can be practically poisoned by malicious actors; and (2) adversaries can compromise language models after poisoning fine-tuning datasets. Our work evaluates for the first time whether language models can also be compromised during pre-training, with a focus on the persistence of pre-training attacks after models are fine-tuned as helpful and harmless chatbots (i.e., after SFT and DPO). We pre-train a series of LLMs from scratch to measure the impact of a potential poisoning adversary under four different attack objectives (denial-of-service, belief manipulation, jailbreaking, and prompt stealing), and across a wide range of model sizes (from 600M to 7B). Our main result is that poisoning only 0.1% of a model's pre-training dataset is sufficient for three out of four attacks to measurably persist through post-training. Moreover, simple attacks like denial-of-service persist through post-training with a poisoning rate of only 0.001%.

Key facts. Cited by 48 papers in the map (data through 14 Oct 2026). Builds on 32 of its 41 references that are in the map. First posted on arXiv 17 Oct 2024.

Builds on

32 of 41 references.

References · 32

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2017Deep reinforcement learning from human preferencescited by 3,912
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  8. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  9. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  10. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  11. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  12. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,055
20 more
  1. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  2. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  3. 2024OLMo: Accelerating the Science of Language Modelscited by 613
  4. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 536
  5. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 529
  6. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 483
  7. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 397
  8. 2020Weight Poisoning Attacks on Pre-trained Modelscited by 341
  9. 2021Multimodal datasets: misogyny, pornography, and malignant stereotypescited by 327
  10. 2023Are aligned neural networks adversarially aligned?cited by 315
  11. 2023Poisoning Web-Scale Training Datasets is Practicalcited by 290
  12. 2023Poisoning Language Models During Instruction Tuningcited by 269
  13. 2024Foundational Challenges in Assuring Alignment and Safety of Large Language Modelscited by 199
  14. 2023Universal Jailbreak Backdoors from Poisoned Human Feedbackcited by 128
  15. 2021Be Careful about Poisoned Word Embeddings: Exploring the Vulnerability of the Embedding Layers in NLP Modelscited by 105
  16. 2023Effective Prompt Extraction from Language Modelscited by 85
  17. 2024BEEAR: Embedding-based Adversarial Removal of Safety Backdoors in Instruction-tuned Language Modelscited by 58
  18. 2024Competition Report: Finding Universal Jailbreak Backdoors in Aligned LLMscited by 28
  19. 2024Dataset and Lessons Learned from the 2024 SaTML LLM Capture-the-Flag Competitioncited by 20
  20. 2020Humpty Dumpty: Controlling Word Meanings via Corpus Poisoningcited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 28; 2026: 20.

48 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  2. 2025Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samplescited by 72
  3. 2025Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-timecited by 43
  4. 2025Position: Adversarial ML for LLMs Is Not Making Any Progresscited by 15
  5. 2025Scalable Fingerprinting of Large Language Modelscited by 14
  6. 2025Cascading Adversarial Bias from Injection to Distillation in Language Modelscited by 9
  7. 2025Gaperon: A Peppered English-French Generative Language Model Suitecited by 9
  8. 2026Emergent and Subliminal Misalignment Through the Lens of Data-Mediated Transfercited by 8
  9. 2025Redirection for Erasing Memory (REM): Towards a universal unlearning method for corrupted datacited by 7
  10. 2026GRP-Obliteration: Unaligning LLMs With a Single Unlabeled Promptcited by 7
  11. 2025Winter Soldier: Backdooring Language Models at Pre-Training with Indirect Data Poisoningcited by 6
  12. 2025Train Once, Answer All: Many Pretraining Experiments for the Cost of Onecited by 6
36 more
  1. 2025Pay Attention to the Triggers: Constructing Backdoors That Survive Distillationcited by 6
  2. 2026Privacy Collapse: Benign Fine-Tuning Can Break Contextual Privacy in Language Modelscited by 6
  3. 2025BitMark: Watermarking Bitwise Autoregressive Image Generative Modelscited by 5
  4. 2025Exploiting Leaderboards for Large-Scale Distribution of Malicious Modelscited by 5
  5. 2025On Code-Induced Reasoning in LLMscited by 5
  6. 2026Infusion: Shaping Model Behavior by Editing Training Data via Influence Functionscited by 5
  7. 2026Human-like autonomy emerges from self-play and a pinch of human datacited by 4
  8. 2025LLMs Can Get "Brain Rot": A Pilot Study on Twitter/Xcited by 3
  9. 2025PoTS: Proof-of-Training-Steps for Backdoor Detection in Large Language Modelscited by 3
  10. 2026OpenSeal: Good, Fast, and Cheap Construction of an Open-Source Southeast Asian LLM via Parallel Datacited by 3
  11. 2026PoisonForge: Task-Level Targeted Poisoning Benchmark for Instruction-Tuned LLMscited by 3
  12. 2025What Really is a Member? Discrediting Membership Inference via Poisoningcited by 2
  13. 2025LLM Hypnosis: Exploiting User Feedback for Unauthorized Knowledge Injection to All Userscited by 2
  14. 2025Prompt Injection Vulnerability of Consensus Generating Applications in Digital Democracycited by 2
  15. 2025Securing Educational LLMs: A Generalised Taxonomy of Attacks on LLMs and DREAD Risk Assessmentcited by 2
  16. 2025FuncPoison: Poisoning Function Library to Hijack Multi-agent Autonomous Driving Systemscited by 2
  17. 2025From Poisoned to Aware: Fostering Backdoor Self-Awareness in LLMscited by 2
  18. 2026When AI reviews science: Can we trust the referee?cited by 2
  19. 2025Poison Once, Refuse Forever: Weaponizing Alignment for Injecting Bias in LLMscited by 1
  20. 2025Locket: Robust Feature-Locking Technique for Language Modelscited by 1
  21. 2025Layer of Truth: Probing Belief Shifts under Continual Pre-Training Poisoningcited by 1
  22. 2026Incentive-Aware AI Safety via Strategic Resource Allocation: A Stackelberg Security Games Perspectivecited by 1
  23. 2026Is a Document Educational or Just Wikipedia-Style? -- Pitfalls of Classifier-Based Quality Filteringcited by 1
  24. 2026RogueMerge: Robust and Unified Attacks against LLM Model Mergingcited by 1
  25. 2026Loss Landscape Poisoning: Targeted Extraction of Unseen Training Data from LLMscited by 1
  26. 2026Safety from Honesty in a Disinterested AI Predictorcited by 1
  27. 2026Isolation as a First-Class Principle for LLM-Agent System Safety: Concepts, Taxonomy, Challenges and Future Directionscited by 1
  28. 2025Can Small-Scale Data Poisoning Exacerbate Dialect-Linked Biases in Large Language Models?cited by 0
  29. 2025BadPromptFL: A Novel Backdoor Threat to Prompt-based Federated Learning in Multimodal Modelscited by 0
  30. 2025On The Dangers of Poisoned LLMs In Security Automationcited by 0
  31. 2026Iterative Finetuning is Mostly Idempotentcited by 0
  32. 2026Sequential Data Poisoning in LLM Post-Trainingcited by 0
  33. 2026Validity Threats for Foundation Model Researchcited by 0
  34. 2026Toward Secure AI-Powered Penetration Testing Agents: Security Threats, Guardrails, and Architectural Perspectivescited by 0
  35. 2026The Privacy Fallacy of Crowdsourced Fine-Tuning: Extracting Proprietary Data via Topic-Based Poisoningcited by 0
  36. 2026Misinformation Without Triggers: From Factual Answers to Downstream Decisionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Multi-Faceted Studies on Data Poisoning can Advance LLM Development
  2. 2025System Prompt Poisoning: Persistent Attacks on Large Language Models Beyond User Injection
  3. 2024Breaking Down the Defenses: A Comparative Survey of Attacks on Large Language Models
  4. 2024PoisonBench: Assessing Large Language Model Vulnerability to Data Poisoning
  5. 2025Multi-Trigger Poisoning Amplifies Backdoor Vulnerabilities in LLMs
  6. 2026PermaFrost-Attack: Stealth Pretraining Seeding(SPS) for planting Logic Landmines During LLM Training
  7. 2024Learning to Poison Large Language Models for Downstream Manipulation
  8. 2025Swallowing the Poison Pills: Insights from Vulnerability Disparity Among LLMs