mlsanityData to 14 Oct 2026

A Theory of Saddle Escape in Deep Nonlinear Networks

arXiv 2605.01288 ↗First posted 2 May 2026cs.LG cond-mat.dis-nn stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

In deep networks with small initialization, training can exhibit long plateaus separated by sharp feature-acquisition transitions. Whereas shallow nonlinear networks and deep linear networks are well studied, extending these analyses to deep nonlinear networks remains challenging. We derive exact scalar and matrix identities for the imbalance of layer weight norms, holding for any smooth activation and any differentiable loss, and use the resulting approximate balance law to control the full finite-width gradient flow through its first escape. This gives a scaling law in the escape time: with a prefix of r layers initialized at the bottleneck scale varepsilon, the escape time is Theta(Gamma-1) for r=1, Theta(Gamma-1log(1/varepsilon)) for r=2, and Theta(Gamma-1varepsilon-(r-2)) for r≥3, where Gamma=|mathbbE[yx]|. We further show that a multimode teacher leaves the saddle once rather than in stages: with scalar output, all mode overlaps grow through a common amplitude, with relative weights fixed in advance. We find close agreement between our theory and experiment.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 51 of its 74 references that are in the map. First posted on arXiv 2 May 2026.

Builds on

51 of 74 references.

References · 51

  1. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  3. 2018On the Spectral Bias of Neural Networkscited by 1,341
  4. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  5. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  6. 2018On Lazy Training in Differentiable Programmingcited by 675
  7. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
  8. 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
  9. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 306
  10. 2017How to Escape Saddle Points Efficientlycited by 285
  11. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 284
  12. 2015Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decompositioncited by 243
38 more
  1. 2017High-dimensional dynamics of generalization error in neural networkscited by 241
  2. 2018A mathematical theory of semantic development in deep neural networkscited by 186
  3. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  4. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 177
  5. 2023SGD learning on neural networks: leap complexity and saddle-to-saddle dynamicscited by 143
  6. 2022The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networkscited by 142
  7. 2023The emergence of clusters in self-attention dynamicscited by 140
  8. 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 127
  9. 2020Online stochastic gradient descent on non-convex losses from high-dimensional inferencecited by 119
  10. 2019Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setupcited by 112
  11. 2020Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamicscited by 96
  12. 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 85
  13. 2017Gradient Descent Can Take Exponential Time to Escape Saddle Pointscited by 74
  14. 2022High-dimensional limit theorems for SGD: Effective dynamics and critical scalingcited by 70
  15. 2018Escaping Saddles with Stochastic Gradientscited by 66
  16. 2019The Implicit Bias of Depth: How Incremental Learning Drives Generalizationcited by 64
  17. 2019Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscapecited by 63
  18. 2023Saddle-to-Saddle Dynamics in Diagonal Linear Networkscited by 57
  19. 2021Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networkscited by 50
  20. 2022Neural networks trained with SGD learn distributions of increasing complexitycited by 50
  21. 2025Training Dynamics of In-Context Learning in Linear Attentioncited by 46
  22. 2023Abide by the Law and Follow the Flow: Conservation Laws for Gradient Flowscited by 45
  23. 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 42
  24. 2022The Neural Race Reduction: Dynamics of Abstraction in Gated Networkscited by 37
  25. 2023On the Stepwise Nature of Self-Supervised Learningcited by 36
  26. 2022Incremental Learning in Diagonal Linear Networkscited by 28
  27. 2019Efficiently escaping saddle points on manifoldscited by 24
  28. 2020Data-Dependence of Plateau Phenomenon in Learning with Neural Network --- Statistical Mechanical Analysiscited by 23
  29. 2025Learning In-context n-grams with Transformers: Sub-n-grams Are Near-stationary Pointscited by 16
  30. 2023Escaping mediocrity: how two-layer networks learn hard generalized linear models with SGDcited by 15
  31. 2025Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networkscited by 14
  32. 2024The Optimization Landscape of SGD Across the Feature Learning Strengthcited by 13
  33. 2024Directional Convergence Near Small Initializations and Saddles in Two-Homogeneous Neural Networkscited by 11
  34. 2024Loss Landscape of Shallow ReLU-like Neural Networks: Stationary Points, Saddle Escape, and Network Embeddingcited by 6
  35. 2024Topological obstruction to the training of shallow ReLU neural networkscited by 6
  36. 2026Topology and Geometry of the Learning Space of ReLU Networks: Connectivity and Singularitiescited by 6
  37. 2021Convergence and Implicit Bias of Gradient Flow on Overparametrized Linear Networkscited by 5
  38. 2025Barriers for Learning in an Evolving World: Mathematical Understanding of Loss of Plasticitycited by 5

50 most cited of 51 All 51 (JSON)

Built on by

Not cited yet.