mlsanityData to 14 Oct 2026

Cascading Through the Hierarchy: Regularizer-Induced Feature Detection as Phase Transitions in Deep Linear Neural Networks

arXiv 2608.06597 ↗First posted 6 Aug 2026cond-mat.stat-mech cond-mat.dis-nn cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

A scientific theory of deep learning, comprising learning dynamics and statistical properties of learned models, is rapidly gaining attention. One of the corner stones of this development are analytically solvable toy models, allowing for the fully tractable analysis of the learning dynamics. Here we analytically investigate such a toy model using the regularization strength as a tunable external parameter - akin to external fields in statistical physics. In previous studies, (i) an onset of learning transition was predicted analytically and (ii) it was phenomenologically/numerically established that tuning the regularization strength can result in a cascade of phase transitions. The number of those transitions was linked to the geometry of the loss landscape determined by the model complexity. Setting up a rigorous framework underpinning the previous numerical observations, our investigation reveals a precise connection between those cascades of phase transitions, learnable features and the underlying geometry. We provide analytic predictions of these phase transitions as well as tractable order parameters related to learned features. At the level of the minimal model, we connect this macroscopic perspective (that can be condensed into an effective description) to the microscopic perspective in terms of the geometry of the loss landscape characterized by the Hessian spectrum. Thus, the presented model provides a platform to explore and sharpen advances made in the scientific theory of deep learning rooted in statistical physics concepts.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 37 of its 67 references that are in the map. First posted on arXiv 6 Aug 2026.

Builds on

37 of 67 references.

References · 37

  1. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  2. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 284
  3. 2017High-dimensional dynamics of generalization error in neural networkscited by 241
  4. 2016Deep Learning without Poor Local Minimacited by 234
  5. 2018A high-bias, low-variance introduction to Machine Learning for physicistscited by 108
  6. 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 85
  7. 2020Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalizationcited by 71
  8. 2019Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizerscited by 61
  9. 2021Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNscited by 53
  10. 2021Analytic Insights into Structure and Rank of Neural Network Hessian Mapscited by 52
  11. 2024From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networkscited by 49
  12. 2024Cautious Optimizers: Improving Training with One Line of Codecited by 47
25 more
  1. 2025Symmetry in Neural Network Parameter Spacescited by 36
  2. 2022Symmetries, flat minima, and the conserved quantities of gradient flowcited by 35
  3. 2019Pure and Spurious Critical Points: a Geometric Study of Linear Networkscited by 33
  4. 2021Power-law escape rate of SGDcited by 29
  5. 2023Implicit bias of SGD in L2-regularized linear DNNs: One-way jumps from high to low rankcited by 29
  6. 2026There Will Be a Scientific Theory of Deep Learningcited by 29
  7. 2023Symmetry Induces Structure and Constraint of Learningcited by 25
  8. 2021The loss landscape of deep linear neural networks: a second-order analysiscited by 24
  9. 2022Exact Solutions of a Deep Linear Networkcited by 24
  10. 2025Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architecturescited by 15
  11. 2025Parameter Symmetry Potentially Unifies Deep Learning Theorycited by 12
  12. 2024Nuclear Norm Regularization for Deep Learningcited by 11
  13. 2024The geometry of the deep linear networkcited by 11
  14. 2025Applications of Statistical Field Theory in Deep Learningcited by 9
  15. 2025An entropy formula for the Deep Linear Networkcited by 8
  16. 2017Exploring the Function Space of Deep-Learning Machinescited by 7
  17. 2023An effective theory of collective deep learningcited by 7
  18. 2025Low-rank bias, weight decay, and model merging in neural networkscited by 5
  19. 2026Noise-Driven Escape from Metastable Phases explains Grokking in Deep Neural Networkscited by 5
  20. 2025Regularization Implies balancedness in the deep linear networkcited by 4
  21. 2025Understanding Learning Invariance in Deep Linear Networkscited by 3
  22. 2024Review and Prospect of Algebraic Research in Equivalent Framework between Statistical Mechanics and Machine Learning Theorycited by 2
  23. 2025Gradient Flow Equations for Deep Linear Neural Networks: A Survey from a Network Perspectivecited by 2
  24. 2025Entropic Regularization in the Deep Linear Networkcited by 2
  25. 2025Phase transitions reveal hierarchical structure in deep neural networkscited by 2

Built on by

Not cited yet.