mlsanityData to 7 Oct 2026

Winter Soldier: Backdooring Language Models at Pre-Training with Indirect Data Poisoning

arXiv 2506.14913 ↗First posted 17 Jun 2025cs.CR cs.LG stat.MLcited by 6

On the map: LLM safety and security → Backdoor and data poisoning attacks

Abstract (arXiv)

The pre-training of large language models (LLMs) relies on massive text datasets sourced from diverse and difficult-to-curate origins. Although membership inference attacks and hidden canaries have been explored to trace data usage, such methods rely on memorization of training data, which LM providers try to limit. In this work, we demonstrate that indirect data poisoning (where the targeted behavior is absent from training data) is not only feasible but also allow to effectively protect a dataset and trace its use. Using gradient-based optimization prompt-tuning, we make a model learn arbitrary secret sequences: secret responses to secret prompts that are absent from the training corpus. We validate our approach on language models pre-trained from scratch and show that less than 0.005% of poisoned tokens are sufficient to covertly make a LM learn a secret and detect it with extremely high confidence (p < 10-55) with a theoretically certifiable scheme. Crucially, this occurs without performance degradation (on LM benchmarks) and despite secrets never appearing in the training set.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security. Builds on 33 of its 38 references that are in the map. First posted on arXiv 17 Jun 2025.

Builds on

33 of 38 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Composite Backdoor Attacks Against Large Language Models11 Oct

Other references · 32

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  4. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  5. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  6. 2020Extracting Training Data from Large Language Modelscited by 2,144
  7. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  8. 2023Detecting Pretraining Data from Large Language Modelscited by 375
  9. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 309
  10. 2022Data Contamination: From Memorization to Exploitationcited by 234
  11. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 233
  12. 2023Proving Test Set Contamination in Black Box Language Modelscited by 227
20 more
  1. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 218
  2. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 216
  3. 2020Auditing Differentially Private Machine Learning: How Private is Private SGD?cited by 213
  4. 2024Do Membership Inference Attacks Work on Large Language Models?cited by 188
  5. 2022Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attackscited by 168
  6. 2021Counterfactual Memorization in Neural Language Modelscited by 155
  7. 2023Privacy Auditing with One (1) Training Runcited by 119
  8. 2024What is in Your Safe Data? Identifying Benign Data that Breaks Safetycited by 114
  9. 2024LLM Dataset Inference: Did you train on my dataset?cited by 107
  10. 2022Untargeted Backdoor Watermark: Towards Harmless and Stealthy Dataset Copyright Protectioncited by 80
  11. 2020Radioactive data: tracing through trainingcited by 67
  12. 2023The Foundation Model Transparency Indexcited by 63
  13. 2024Membership Inference Attacks Cannot Prove that a Model Was Trained On Your Datacited by 49
  14. 2024Persistent Pre-Training Poisoning of LLMscited by 46
  15. 2024Proving membership in LLM pretraining data via data watermarkscited by 38
  16. 2023Did You Train on My Dataset? Towards Public Dataset Protection with Clean-Label Backdoor Watermarkingcited by 37
  17. 2024Copyright Traps for Large Language Modelscited by 36
  18. 2024The Stronger the Diffusion Model, the Easier the Backdoor: Data Poisoning to Induce Copyright Breaches Without Adjusting Finetuning Pipelinecited by 33
  19. 2023MoPe: Model Perturbation-based Privacy Attacks on Language Modelscited by 25
  20. 2024Data Taggants: Dataset Ownership Verification via Harmless Targeted Data Poisoningcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 3.

6 papers.

  1. 2025Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samplescited by 66
  2. 2025BadThink: Triggered Overthinking Attacks on Chain-of-Thought Reasoning in Large Language Modelscited by 7
  3. 2026Byzantine Machine Learning: MultiKrum and an optimal notion of robustnesscited by 1
  4. 2025Perturb Your Data: Paraphrase-Guided Training Data Watermarkingcited by 0
  5. 2026Pretraining Data Can Be Poisoned through Computational Propagandacited by 0
  6. 2026The Privacy Fallacy of Crowdsourced Fine-Tuning: Extracting Proprietary Data via Topic-Based Poisoningcited by 0