# You Are What You Eat -- AI Alignment Requires Understanding How Data Shapes Structure and Generalisation

arXiv: https://arxiv.org/abs/2502.05475 · HTML: https://mlsanity.com/paper/2502.05475/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Feb 2025.

## Built on by (most cited first, 17 of 17)

- [Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometry](https://mlsanity.com/paper/2503.01822/) (2025)
- [Bayesian Influence Functions for Hessian-Free Data Attribution](https://mlsanity.com/paper/2509.26544/) (2025)
- [Value Drifts: Tracing Value Alignment During LLM Post-Training](https://mlsanity.com/paper/2510.26707/) (2025)
- [The Loss Kernel: A Geometric Probe for Deep Learning Interpretability](https://mlsanity.com/paper/2509.26537/) (2025)
- [Influence Dynamics and Stagewise Data Attribution](https://mlsanity.com/paper/2510.12071/) (2025)
- [A Mathematical Philosophy of Explanations in Mechanistic Interpretability -- The Strange Science Part I.i](https://mlsanity.com/paper/2505.00808/) (2025)
- [Stagewise Reinforcement Learning and the Geometry of the Regret Landscape](https://mlsanity.com/paper/2601.07524/) (2026)
- [Mechanistic Anomaly Detection via Functional Attribution](https://mlsanity.com/paper/2604.18970/) (2026)
- [Interpreting Reinforcement Learning Agents with Susceptibilities](https://mlsanity.com/paper/2605.08007/) (2026)
- [Radical AI Interpretability](https://mlsanity.com/paper/2606.26523/) (2026)
- [Interfacial and bulk switching MoS2 memristors for an all-2D reservoir computing framework](https://mlsanity.com/paper/2511.16557/) (2025)
- [Using physics-inspired Singular Learning Theory to understand grokking & other phase transitions in modern neural networks](https://mlsanity.com/paper/2512.00686/) (2025)
- [Stochastic Gradient Descent in the Saddle-to-Saddle Regime of Deep Linear Networks](https://mlsanity.com/paper/2604.06366/) (2026)
- [Phase Transitions in Driven Informational Systems: A Two-Field Perspective on Learning Theory and Non-Equilibrium Chemistry](https://mlsanity.com/paper/2605.16325/) (2026)
- [Structure and Scale in Simplicial Sequence Modelling](https://mlsanity.com/paper/2606.01302/) (2026)
- [Critical Percolation as a Synthetic Data Model for Interpretability](https://mlsanity.com/paper/2606.20347/) (2026)
- [Math for AI safety: an invitation for mathematicians](https://mlsanity.com/paper/2609.15289/) (2026)

## Abstract (arXiv)

In this position paper, we argue that understanding the relation between structure in the data distribution and structure in trained models is central to AI alignment. First, we discuss how two neural networks can have equivalent performance on the training set but compute their outputs in essentially different ways and thus generalise differently. For this reason, standard testing and evaluation are insufficient for obtaining assurances of safety for widely deployed generally intelligent systems. We argue that to progress beyond evaluation to a robust mathematical science of AI alignment, we need to develop statistical foundations for an understanding of the relation between structure in the data distribution, internal structure in models, and how these structures underlie generalisation.