mlsanityData to 7 Oct 2026

Saddle-To-Saddle Dynamics in Deep ReLU Networks: Low-Rank Bias in the First Saddle Escape

arXiv 2505.21722 ↗First posted 27 May 2025cs.LG cs.AI stat.MLcited by 4

Not on the map. Why?

Abstract (arXiv)

When a deep ReLU network is initialized with small weights, gradient descent (GD) is at first dominated by the saddle at the origin in parameter space. We study the so-called escape directions along which GD leaves the origin, which play a similar role as the eigenvectors of the Hessian for strict saddles. We show that the optimal escape direction features a low-rank bias in its deeper layers: the first singular value of the ℓ-th layer weight matrix is at least ℓ^frac14 larger than any other singular value. We also prove a number of related results about these escape directions. We suggest that deep ReLU networks exhibit saddle-to-saddle dynamics, with GD visiting a sequence of saddles with increasing bottleneck rank (Jacot, 2023).

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 47 of its 55 references that are in the map. First posted on arXiv 27 May 2025.

Builds on

47 of 55 references.

References · 47

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  3. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  4. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  5. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  6. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  7. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  8. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  9. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  10. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 329
  11. 2017Implicit Regularization in Matrix Factorizationcited by 318
  12. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
35 more
  1. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  2. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 206
  3. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  4. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 183
  5. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  6. 2018A mathematical theory of semantic development in deep neural networkscited by 182
  7. 2020Feature Learning in Infinite-Width Neural Networkscited by 182
  8. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 151
  9. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
  10. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 148
  11. 2022The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networkscited by 141
  12. 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 125
  13. 2019Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networkscited by 122
  14. 2019A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Casecited by 113
  15. 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 107
  16. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 104
  17. 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 74
  18. 2020Phase diagram for two-layer ReLU neural networks at infinite-width limitcited by 65
  19. 2021The staircase property: How hierarchical structure can guide deep learningcited by 62
  20. 2023Saddle-to-Saddle Dynamics in Diagonal Linear Networkscited by 54
  21. 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 41
  22. 2019The Barron Space and the Flow-induced Function Spaces for Neural Network Modelscited by 39
  23. 2022Implicit Bias of Large Depth Networks: a Notion of Rank for Nonlinear Functionscited by 38
  24. 2023On the Stepwise Nature of Self-Supervised Learningcited by 36
  25. 2022Label noise (stochastic) gradient descent implicitly solves the Lasso for quadratic parametrisationcited by 29
  26. 2023Implicit bias of SGD in L2-regularized linear DNNs: One-way jumps from high to low rankcited by 28
  27. 2019The asymptotic spectrum of the Hessian of DNN throughout trainingcited by 19
  28. 2024Mixed Dynamics In Linear Networks: Unifying the Lazy and Active Regimescited by 18
  29. 2023Bottleneck Structure in Learned Features: Low-Dimension vs Regularity Tradeoffcited by 17
  30. 2024Early alignment in two-layer networks training is a two-edged swordcited by 17
  31. 2024The Optimization Landscape of SGD Across the Feature Learning Strengthcited by 13
  32. 2024Which Frequencies do CNNs Need? Emergent Bottleneck Structure in Feature Learningcited by 9
  33. 2022Embedding Principle in Depth for the Loss Landscape Analysis of Deep Neural Networkscited by 7
  34. 2024When Are Bias-Free ReLU Networks Effectively Linear Networks?cited by 3
  35. 2024Hamiltonian Mechanics of Feature Learning: Bottleneck Structure in Leaky ResNetscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architecturescited by 14
  2. 2025Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networkscited by 11
  3. 2026A Theory of Saddle Escape in Deep Nonlinear Networkscited by 0
  4. 2026Gradient Flow Dynamics and Implicit Bias of Diagonal Linear Networks under Infinitesimal Initializationcited by 0