# Phase Transitions between Accuracy Regimes in L2 regularized Deep Neural Networks

arXiv: https://arxiv.org/abs/2505.06597 · HTML: https://mlsanity.com/paper/2505.06597/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 20 references that are in the map.
- First posted on arXiv 10 May 2025.

## Builds on (most cited first, 9 of 9)

- [Identifying and attacking the saddle point problem in high-dimensional non-convex optimization](https://mlsanity.com/paper/1406.2572/) (2014)
- [The Loss Surfaces of Multilayer Networks](https://mlsanity.com/paper/1412.0233/) (2014)
- [On the geometry of generalization and memorization in deep neural networks](https://mlsanity.com/paper/2105.14602/) (2021)
- [The Global Landscape of Neural Networks: An Overview](https://mlsanity.com/paper/2007.01429/) (2020)
- [Grokking as a First Order Phase Transition in Two Layer Networks](https://mlsanity.com/paper/2310.03789/) (2023)
- [Learnability for the Information Bottleneck](https://mlsanity.com/paper/1907.07331/) (2019)
- [The Complexity Dynamics of Grokking](https://mlsanity.com/paper/2412.09810/) (2024)
- [Feature Learning in L2-regularized DNNs: Attraction/Repulsion and Sparsity](https://mlsanity.com/paper/2205.15809/) (2022)
- [Phase Transitions for the Information Bottleneck in Representation Learning](https://mlsanity.com/paper/2001.01878/) (2020)

## Abstract (arXiv)

Increasing the L2 regularization of Deep Neural Networks (DNNs) causes a first-order phase transition into the under-parametrized phase -- the so-called onset-of learning. We explain this transition via the scalar (Ricci) curvature of the error landscape. We predict new transition points as the data complexity is increased and, in accordance with the theory of phase transitions, the existence of hysteresis effects. We confirm both predictions numerically. Our results provide a natural explanation of the recently discovered phenomenon of 'grokking' as DNN models getting stuck in a local minimum of the error surface, corresponding to a lower accuracy phase. Our work paves the way for new probing methods of the intrinsic structure of DNNs in and beyond the L2 context.