mlsanityData to 7 Oct 2026

Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsity

arXiv 2106.15933 ↗First posted 30 Jun 2021stat.ML cs.LGcited by 83

Not on the map. Why?

Abstract (arXiv)

The dynamics of Deep Linear Networks (DLNs) is dramatically affected by the variance σ2 of the parameters at initialization θ0. For DLNs of width w, we show a phase transition w.r.t. the scaling γ of the variance σ2=w-γ as w→∞: for large variance (γ<1), θ0 is very close to a global minimum but far from any saddle point, and for small variance (γ>1), θ0 is close to a saddle point and far from any global minimum. While the first case corresponds to the well-studied NTK regime, the second case is less understood. This motivates the study of the case γ → +∞, where we conjecture a Saddle-to-Saddle dynamics: throughout training, gradient descent visits the neighborhoods of a sequence of saddles, each corresponding to linear maps of increasing rank, until reaching a sparse global minimum. We support this conjecture with a theorem for the dynamics between the first two saddles, as well as some numerical experiments.

Key facts. Cited by 83 papers in the map (data through 7 Oct 2026). Builds on 36 of its 48 references that are in the map. First posted on arXiv 30 Jun 2021.

Builds on

36 of 48 references.

References · 36

  1. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  2. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  3. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  4. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  5. 2017Deep Neural Networks as Gaussian Processescited by 617
  6. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  7. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  8. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  9. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  10. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
  11. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
  12. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
24 more
  1. 2017How to Escape Saddle Points Efficientlycited by 284
  2. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  3. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 259
  4. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  5. 2017High-dimensional dynamics of generalization error in neural networkscited by 238
  6. 2016Deep Learning without Poor Local Minimacited by 233
  7. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  8. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  9. 2018A mathematical theory of semantic development in deep neural networkscited by 182
  10. 2020Feature Learning in Infinite-Width Neural Networkscited by 182
  11. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 180
  12. 2020Directional convergence and alignment in deep learningcited by 172
  13. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 148
  14. 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 146
  15. 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 125
  16. 2019Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networkscited by 122
  17. 2019Dynamics of Deep Neural Networks and Neural Tangent Hierarchycited by 96
  18. 2017Gradient Descent Can Take Exponential Time to Escape Saddle Pointscited by 74
  19. 2020A Unifying View on Implicit Bias in Training Linear Neural Networkscited by 74
  20. 2020Implicit Bias in Deep Linear Classification: Initialization Scale vs Training Accuracycited by 70
  21. 2019The Implicit Bias of Depth: How Incremental Learning Drives Generalizationcited by 64
  22. 2020Training Linear Neural Networks: Non-Local Convergence and Complexity Resultscited by 19
  23. 2018Learning Deep Models: Critical Points and Local Opennesscited by 10
  24. 2019Disentangling feature and lazy training in deep neural networkscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 7; 2023: 18; 2024: 21; 2025: 15; 2026: 21.

83 papers.

Sort:

By area:

  1. 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 74
  2. 2023Generalization on the Unseen, Logic Reasoning and Degree Curriculumcited by 73
  3. 2023How Two-Layer Neural Networks Learn, One (Giant) Step at a Timecited by 67
  4. 2022SGD with Large Step Sizes Learns Sparse Featurescited by 66
  5. 2023Saddle-to-Saddle Dynamics in Diagonal Linear Networkscited by 54
  6. 2023On Learning Gaussian Multi-index Models with Gradient Flowcited by 53
  7. 2023Transformers learn through gradual rank increasecited by 52
  8. 2024From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networkscited by 46
  9. 2024Dynamic metastability in the self-attention modelcited by 42
  10. 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 41
  11. 2022Implicit Bias of Large Depth Networks: a Notion of Rank for Nonlinear Functionscited by 38
  12. 2023On the Stepwise Nature of Self-Supervised Learningcited by 36
38 more
  1. 2023SGD Finds then Tunes Features in Two-Layer Neural Networks with near-Optimal Sample Complexity: A Case Study in the XOR problemcited by 36
  2. 2021ZerO Initialization: Initializing Neural Networks with only Zeros and Onescited by 28
  3. 2023Implicit bias of SGD in L2-regularized linear DNNs: One-way jumps from high to low rankcited by 28
  4. 2023Dynamical versus Bayesian Phase Transitions in a Toy Model of Superpositioncited by 28
  5. 2022Incremental Learning in Diagonal Linear Networkscited by 27
  6. 2025The emergence of sparse attention: impact of data distribution and benefits of repetitioncited by 25
  7. 2026There Will Be a Scientific Theory of Deep Learningcited by 21
  8. 2022Infinite-width limit of deep linear neural networkscited by 20
  9. 2023Understanding Unimodal Bias in Multimodal Deep Linear Networkscited by 18
  10. 2024Mixed Dynamics In Linear Networks: Unifying the Lazy and Active Regimescited by 18
  11. 2022Feature Learning in L2-regularized DNNs: Attraction/Repulsion and Sparsitycited by 17
  12. 2024Deep linear networks for regression are implicitly regularized towards flat minimacited by 17
  13. 2024Synchronization on circles and spheres with nonlinear interactionscited by 17
  14. 2023Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputscited by 16
  15. 2023Understanding Multi-phase Optimization Dynamics and Rich Nonlinear Behaviors of ReLU Networkscited by 15
  16. 2025Learning In-context n-grams with Transformers: Sub-n-grams Are Near-stationary Pointscited by 15
  17. 2024Learning Associative Memories with Gradient Descentcited by 14
  18. 2025Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architecturescited by 14
  19. 2024The Optimization Landscape of SGD Across the Feature Learning Strengthcited by 13
  20. 2025Adaptive kernel predictors from feature-learning infinite limits of neural networkscited by 13
  21. 2023On the different regimes of Stochastic Gradient Descentcited by 12
  22. 2024Sliding down the stairs: how correlated latent variables accelerate learning with neural networkscited by 12
  23. 2024The Persistence of Neural Collapse Despite Low-Rank Biascited by 12
  24. 2025Dynamics of Transient Structure in In-Context Linear Regression Transformerscited by 12
  25. 2025Parameter Symmetry Potentially Unifies Deep Learning Theorycited by 11
  26. 2025Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networkscited by 11
  27. 2023InRank: Incremental Low-Rank Learningcited by 10
  28. 2024Only Strict Saddles in the Energy Landscape of Predictive Coding Networks?cited by 10
  29. 2023Type-II Saddles and Probabilistic Stability of Stochastic Gradient Descentcited by 8
  30. 2023The star-shaped space of solutions of the spherical negative perceptroncited by 8
  31. 2025Position: Solve Layerwise Linear Models First to Understand Neural Dynamical Phenomena (Neural Collapse, Emergence, Lazy/Rich Regime, and Grokking)cited by 8
  32. 2025A Tale of Two Symmetries: Exploring the Loss Landscape of Equivariant Modelscited by 8
  33. 2026Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retentioncited by 8
  34. 2025Closed-Form Training Dynamics Reveal Learned Features and Linear Structure in Word2Vec-like Modelscited by 7
  35. 2026Sequential Group Composition: A Window into the Mechanics of Deep Learningcited by 7
  36. 2026To Use or not to Use Muon: How Simplicity Bias in Optimizers Matterscited by 7
  37. 2024Estimating the Local Learning Coefficient at Scalecited by 6
  38. 2024Early Directional Convergence in Deep Homogeneous Neural Networks for Small Initializationscited by 6

50 most cited of 83 All 83 (JSON)