mlsanityData to 14 Oct 2026

Directional Convergence Near Small Initializations and Saddles in Two-Homogeneous Neural Networks

arXiv 2402.09226 ↗First posted 14 Feb 2024cs.LG math.OC stat.MLcited by 11

Not on the map. Why?

Abstract (arXiv)

This paper examines gradient flow dynamics of two-homogeneous neural networks for small initializations, where all weights are initialized near the origin. For both square and logistic losses, it is shown that for sufficiently small initializations, the gradient flow dynamics spend sufficient time in the neighborhood of the origin to allow the weights of the neural network to approximately converge in direction to the Karush-Kuhn-Tucker (KKT) points of a neural correlation function that quantifies the correlation between the output of the neural network and corresponding labels in the training data set. For square loss, it has been observed that neural networks undergo saddle-to-saddle dynamics when initialized close to the origin. Motivated by this, this paper also shows a similar directional convergence among weights of small magnitude in the neighborhood of certain saddle points.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 24 of its 34 references that are in the map. First posted on arXiv 14 Feb 2024.

Builds on

24 of 34 references.

References · 24

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2018On Lazy Training in Differentiable Programmingcited by 675
  3. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  4. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  5. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
  6. 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
  7. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 342
  8. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
  9. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 306
  10. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  11. 2020Directional convergence and alignment in deep learningcited by 178
  12. 2018Stochastic subgradient method converges on tame functionscited by 124
12 more
  1. 2021Neural Networks as Kernel Learners: The Silent Alignment Effectcited by 106
  2. 2021Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Biascited by 83
  3. 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 76
  4. 2020Phase diagram for two-layer ReLU neural networks at infinite-width limitcited by 67
  5. 2023Saddle-to-Saddle Dynamics in Diagonal Linear Networkscited by 57
  6. 2019Gradient Dynamics of Shallow Univariate ReLU Networkscited by 46
  7. 2023Understanding Incremental Learning of Gradient Descent: A Fine-grained Analysis of Matrix Sensingcited by 42
  8. 2023Early Neuron Alignment in Two-layer ReLU Networks with Small Initializationcited by 28
  9. 2022On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Biascited by 21
  10. 2023Phase Diagram of Initial Condensation for Two-layer Neural Networkscited by 16
  11. 2023Understanding Multi-phase Optimization Dynamics and Rich Nonlinear Behaviors of ReLU Networkscited by 15
  12. 2022Early Stage Convergence and Global Convergence of Training Mildly Parameterized Neural Networkscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 5; 2026: 2.

11 papers.

  1. 2025An overview of condensation phenomenon in deep learningcited by 17
  2. 2025Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architecturescited by 15
  3. 2025Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networkscited by 14
  4. 2024Simplicity bias and optimization threshold in two-layer ReLU networkscited by 11
  5. 2024Early Directional Convergence in Deep Homogeneous Neural Networks for Small Initializationscited by 7
  6. 2024Can Implicit Bias Imply Adversarial Robustness?cited by 5
  7. 2025SAD Neural Networks: Divergent Gradient Flows and Asymptotic Optimality via o-minimal Structurescited by 5
  8. 2025Neural Collapse under Gradient Flow on Shallow ReLU Networks for Orthogonally Separable Datacited by 2
  9. 2024On Multi-Stage Loss Dynamics in Neural Networks: Mechanisms of Plateau and Descent Stagescited by 1
  10. 2026A Theory of Saddle Escape in Deep Nonlinear Networkscited by 0
  11. 2026Weight Decay and Neuron Condensation: A Three-Stage Analysis of Two-Layer ReLU Networkscited by 0