mlsanityData to 7 Oct 2026

Early Directional Convergence in Deep Homogeneous Neural Networks for Small Initializations

arXiv 2403.08121 ↗First posted 12 Mar 2024cs.LG math.OC stat.MLcited by 6

Not on the map. Why?

Abstract (arXiv)

This paper studies the gradient flow dynamics that arise when training deep homogeneous neural networks assumed to have locally Lipschitz gradients and an order of homogeneity strictly greater than two. It is shown here that for sufficiently small initializations, during the early stages of training, the weights of the neural network remain small in (Euclidean) norm and approximately converge in direction to the Karush-Kuhn-Tucker (KKT) points of the recently introduced neural correlation function. Additionally, this paper also studies the KKT points of the neural correlation function for feed-forward networks with (Leaky) ReLU and polynomial (Leaky) ReLU activations, deriving necessary and sufficient conditions for rank-one KKT points.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 33 of its 40 references that are in the map. First posted on arXiv 12 Mar 2024.

Builds on

33 of 40 references.

References · 33

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  4. 2018On Lazy Training in Differentiable Programmingcited by 671
  5. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  6. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  7. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  8. 2017Implicit Regularization in Matrix Factorizationcited by 318
  9. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
  10. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  11. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  12. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
21 more
  1. 2020Directional convergence and alignment in deep learningcited by 172
  2. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 148
  3. 2017Theoretical insights into the optimization landscape of over-parameterized shallow neural networkscited by 128
  4. 2021Neural Networks as Kernel Learners: The Silent Alignment Effectcited by 106
  5. 2017Recovery Guarantees for One-hidden-layer Neural Networkscited by 94
  6. 2016Approximation by Combinations of ReLU and Squared ReLU Ridge Functions with ℓ1 and ℓ0 Controlscited by 84
  7. 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 83
  8. 2021Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Biascited by 82
  9. 2021Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstructioncited by 77
  10. 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 74
  11. 2014On the Computational Efficiency of Training Neural Networkscited by 71
  12. 2019Conservative set valued fields, automatic differentiation, stochastic gradient method and deep learningcited by 71
  13. 2020Phase diagram for two-layer ReLU neural networks at infinite-width limitcited by 65
  14. 2018Gradient Descent Quantizes ReLU Network Featurescited by 57
  15. 2019Approximation spaces of deep neural networkscited by 51
  16. 2023Understanding Incremental Learning of Gradient Descent: A Fine-grained Analysis of Matrix Sensingcited by 40
  17. 2023Early Neuron Alignment in Two-layer ReLU Networks with Small Initializationcited by 28
  18. 2020Convergence of constant step stochastic gradient descent for non-smooth non-convex functionscited by 27
  19. 2022Empirical Phase Diagram for Three-layer Neural Networks with Infinite Widthcited by 17
  20. 2023Understanding Multi-phase Optimization Dynamics and Rich Nonlinear Behaviors of ReLU Networkscited by 15
  21. 2024Directional Convergence Near Small Initializations and Saddles in Two-Homogeneous Neural Networkscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 1.

6 papers.

  1. 2025An overview of condensation phenomenon in deep learningcited by 13
  2. 2025Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networkscited by 11
  3. 2025From Condensation to Rank Collapse: A Two-Stage Analysis of Transformer Training Dynamicscited by 9
  4. 2026The Weight Gram Matrix Captures Sequential Feature Linearization in Deep Networkscited by 3
  5. 2024Towards Better Generalization: Weight Decay Induces Low-rank Bias for Neural Networkscited by 2
  6. 2024On Multi-Stage Loss Dynamics in Neural Networks: Mechanisms of Plateau and Descent Stagescited by 0