# Cascading Through the Hierarchy: Regularizer-Induced Feature Detection as Phase Transitions in Deep Linear Neural Networks

arXiv: https://arxiv.org/abs/2608.06597 · HTML: https://mlsanity.com/paper/2608.06597/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 37 of its 67 references that are in the map.
- First posted on arXiv 6 Aug 2026.

## Builds on (most cited first, 30 of 37)

- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization](https://mlsanity.com/paper/1802.06509/) (2018)
- [High-dimensional dynamics of generalization error in neural networks](https://mlsanity.com/paper/1710.03667/) (2017)
- [Deep Learning without Poor Local Minima](https://mlsanity.com/paper/1605.07110/) (2016)
- [A high-bias, low-variance introduction to Machine Learning for physicists](https://mlsanity.com/paper/1803.08823/) (2018)
- [Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsity](https://mlsanity.com/paper/2106.15933/) (2021)
- [Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalization](https://mlsanity.com/paper/2012.04030/) (2020)
- [Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizers](https://mlsanity.com/paper/1910.05505/) (2019)
- [Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNs](https://mlsanity.com/paper/2112.15383/) (2021)
- [Analytic Insights into Structure and Rank of Neural Network Hessian Maps](https://mlsanity.com/paper/2106.16225/) (2021)
- [From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks](https://mlsanity.com/paper/2409.14623/) (2024)
- [Cautious Optimizers: Improving Training with One Line of Code](https://mlsanity.com/paper/2411.16085/) (2024)
- [Symmetry in Neural Network Parameter Spaces](https://mlsanity.com/paper/2506.13018/) (2025)
- [Symmetries, flat minima, and the conserved quantities of gradient flow](https://mlsanity.com/paper/2210.17216/) (2022)
- [Pure and Spurious Critical Points: a Geometric Study of Linear Networks](https://mlsanity.com/paper/1910.01671/) (2019)
- [Power-law escape rate of SGD](https://mlsanity.com/paper/2105.09557/) (2021)
- [Implicit bias of SGD in L2-regularized linear DNNs: One-way jumps from high to low rank](https://mlsanity.com/paper/2305.16038/) (2023)
- [There Will Be a Scientific Theory of Deep Learning](https://mlsanity.com/paper/2604.21691/) (2026)
- [Symmetry Induces Structure and Constraint of Learning](https://mlsanity.com/paper/2309.16932/) (2023)
- [The loss landscape of deep linear neural networks: a second-order analysis](https://mlsanity.com/paper/2107.13289/) (2021)
- [Exact Solutions of a Deep Linear Network](https://mlsanity.com/paper/2202.04777/) (2022)
- [Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architectures](https://mlsanity.com/paper/2512.20607/) (2025)
- [Parameter Symmetry Potentially Unifies Deep Learning Theory](https://mlsanity.com/paper/2502.05300/) (2025)
- [Nuclear Norm Regularization for Deep Learning](https://mlsanity.com/paper/2405.14544/) (2024)
- [The geometry of the deep linear network](https://mlsanity.com/paper/2411.09004/) (2024)
- [Applications of Statistical Field Theory in Deep Learning](https://mlsanity.com/paper/2502.18553/) (2025)
- [An entropy formula for the Deep Linear Network](https://mlsanity.com/paper/2509.09088/) (2025)
- [Exploring the Function Space of Deep-Learning Machines](https://mlsanity.com/paper/1708.01422/) (2017)
- [An effective theory of collective deep learning](https://mlsanity.com/paper/2310.12802/) (2023)
- [Low-rank bias, weight decay, and model merging in neural networks](https://mlsanity.com/paper/2502.17340/) (2025)

## Abstract (arXiv)

A scientific theory of deep learning, comprising learning dynamics and statistical properties of learned models, is rapidly gaining attention. One of the corner stones of this development are analytically solvable toy models, allowing for the fully tractable analysis of the learning dynamics. Here we analytically investigate such a toy model using the regularization strength as a tunable external parameter - akin to external fields in statistical physics. In previous studies, (i) an onset of learning transition was predicted analytically and (ii) it was phenomenologically/numerically established that tuning the regularization strength can result in a cascade of phase transitions. The number of those transitions was linked to the geometry of the loss landscape determined by the model complexity. Setting up a rigorous framework underpinning the previous numerical observations, our investigation reveals a precise connection between those cascades of phase transitions, learnable features and the underlying geometry. We provide analytic predictions of these phase transitions as well as tractable order parameters related to learned features. At the level of the minimal model, we connect this macroscopic perspective (that can be condensed into an effective description) to the microscopic perspective in terms of the geometry of the loss landscape characterized by the Hessian spectrum. Thus, the presented model provides a platform to explore and sharpen advances made in the scientific theory of deep learning rooted in statistical physics concepts.