# Using physics-inspired Singular Learning Theory to understand grokking & other phase transitions in modern neural networks

arXiv: https://arxiv.org/abs/2512.00686 · HTML: https://mlsanity.com/paper/2512.00686/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 6 of its 14 references that are in the map.
- First posted on arXiv 30 Nov 2025.

## Builds on (most cited first, 6 of 6)

- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022)
- [The Local Learning Coefficient: A Singularity-Aware Complexity Measure](https://mlsanity.com/paper/2308.12108/) (2023)
- [Deep Learning is Singular, and That's Good](https://mlsanity.com/paper/2010.11560/) (2020)
- [Dynamical versus Bayesian Phase Transitions in a Toy Model of Superposition](https://mlsanity.com/paper/2310.06301/) (2023)
- [You Are What You Eat -- AI Alignment Requires Understanding How Data Shapes Structure and Generalisation](https://mlsanity.com/paper/2502.05475/) (2025)

## Abstract (arXiv)

Classical statistical inference and learning theory often fail to explain the success of modern neural networks. A key reason is that these models are non-identifiable (singular), violating core assumptions behind PAC bounds and asymptotic normality. Singular learning theory (SLT), a physics-inspired framework grounded in algebraic geometry, has gained popularity for its ability to close this theory-practice gap. In this paper, we empirically study SLT in toy settings relevant to interpretability and phase transitions. First, we understand the SLT free energy Fn by testing an Arrhenius-style rate hypothesis using both a grokking modulo-arithmetic model and Anthropic's Toy Models of Superposition. Second, we understand the local learning coefficient λα by measuring how it scales with problem difficulty across several controlled network families (polynomial regressors, low-rank linear networks, and low-rank autoencoders). Our experiments recover known scaling laws while others yield meaningful deviations from theoretical expectations. Overall, our paper illustrates the many merits of SLT for understanding neural network phase transitions, and poses open research questions for the field.