# A Solvable Theory of Pre-training Data Poisoning: Regime-Dependent Scaling Exponents

arXiv: https://arxiv.org/abs/2609.32288 · HTML: https://mlsanity.com/paper/2609.32288/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security.
- Builds on 26 of its 63 references that are in the map.
- First posted on arXiv 26 Sep 2026.

## Placed in

- LLM safety and security / [Backdoor and data poisoning attacks](https://mlsanity.com/safety/backdoor-and-data-poisoning-attacks/)

## Builds on (most cited first, 26 of 26)

- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training](https://mlsanity.com/paper/2401.05566/) (2024)
- [Explaining Neural Scaling Laws](https://mlsanity.com/paper/2102.06701/) (2021)
- [Poisoning Web-Scale Training Datasets is Practical](https://mlsanity.com/paper/2302.10149/) (2023)
- [Stronger Data Poisoning Attacks Break Data Sanitization Defenses](https://mlsanity.com/paper/1811.00741/) (2018)
- [A Solvable Model of Neural Scaling Laws](https://mlsanity.com/paper/2210.16859/) (2022)
- [Learning Curve Theory](https://mlsanity.com/paper/2102.04074/) (2021)
- [Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples](https://mlsanity.com/paper/2510.07192/) (2025)
- [How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Model](https://mlsanity.com/paper/2307.02129/) (2023)
- [A Neural Scaling Law from the Dimension of the Data Manifold](https://mlsanity.com/paper/2004.10802/) (2020)
- [Safety Pretraining: Toward the Next Generation of Safe AI](https://mlsanity.com/paper/2504.16980/) (2025)
- [Towards a theory of how the structure of language is acquired by deep neural networks](https://mlsanity.com/paper/2406.00048/) (2024)
- [Scaling Trends for Data Poisoning in LLMs](https://mlsanity.com/paper/2408.02946/) (2024)
- [Symmetry in language statistics shapes the geometry of model representations](https://mlsanity.com/paper/2602.15029/) (2026)
- [Deriving Neural Scaling Laws from the statistics of natural language](https://mlsanity.com/paper/2602.07488/) (2026)
- [On the origin of neural scaling laws: from random graphs to natural language](https://mlsanity.com/paper/2601.10684/) (2026)
- [A Defense of the Quadratic Model](https://mlsanity.com/paper/2607.21716/) (2026)
- [On the Emergence of Linear Analogies in Word Embeddings](https://mlsanity.com/paper/2505.18651/) (2025)
- [Universal One-third Time Scaling in Learning Peaked Distributions](https://mlsanity.com/paper/2602.03685/) (2026)
- [Closed-Form Training Dynamics Reveal Learned Features and Linear Structure in Word2Vec-like Models](https://mlsanity.com/paper/2502.09863/) (2025)
- [A Latent Variable Model Approach to PMI-based Word Embeddings](https://mlsanity.com/paper/1502.03520/) (2015)
- [When Should We Introduce Safety Interventions During Pretraining?](https://mlsanity.com/paper/2601.07087/) (2026)
- [Scaling Laws Revisited: Modeling the Role of Data Quality in Language Model Pretraining](https://mlsanity.com/paper/2510.03313/) (2025)
- [Symmetry Breaking in Transformers for Efficient and Interpretable Training](https://mlsanity.com/paper/2601.22257/) (2026)
- [Deep networks learn to parse uniform-depth context-free languages from local statistics](https://mlsanity.com/paper/2602.06065/) (2026)
- [Jailbreak Scaling Laws for Large Language Models: Polynomial-Exponential Crossover](https://mlsanity.com/paper/2603.11331/) (2026)

## Abstract (arXiv)

Pre-training data poisoning of large language models is usually studied using targeted backdoors and their survival through safety post-training, which leaves open a more basic question: how does a model's clean data performance degrade as the poison rate varepsilon grows? Motivated by our controlled pre-training runs of OLMo-style models, in which the relative clean data validation perplexity increase Delta between poisoned and clean models matched in architecture, token budget, and optimization schedule is well fit by a power law Delta≈ Cvarepsilona with a non-integer exponent, we ask what such a law requires theoretically. We first prove an analyticity barrier: whenever the contaminated objective depends analytically on varepsilon around a nondegenerate clean data optimum, Delta is generically quadratic in varepsilon, so a generic non-integer exponent is a signature of genuinely singular structure. We then supply that structure in solvable truncated ridge regression with heavy-tailed covariates, controlled by q_star, and a label-shift poisoning. Our central result is that the excess risk scaling exponent depending on the order of limits: in the higher dimensional proportional regime it is εq_star/(q_star+2), whereas taking the ample-data limit first gives ε2-2/q_star, and the limits do not commute. We confirm this prediction through several numerical simulations. Finally, we argue that finite training time plays the role of truncation on the curvature spectrum in local LLM pre-training, deriving the observed scaling law under heavy tailed inverse curvature spectrum as a modeling hypothesis.