mlsanityData to 7 Oct 2026

Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputs

arXiv 2206.00939 ↗First posted 2 Jun 2022stat.ML cs.LGcited by 74

Not on the map. Why?

Abstract (arXiv)

The training of neural networks by gradient descent methods is a cornerstone of the deep learning revolution. Yet, despite some recent progress, a complete theory explaining its success is still missing. This article presents, for orthogonal input vectors, a precise description of the gradient flow dynamics of training one-hidden layer ReLU neural networks for the mean squared error at small initialisation. In this setting, despite non-convexity, we show that the gradient flow converges to zero loss and characterise its implicit bias towards minimum variation norm. Furthermore, some interesting phenomena are highlighted: a quantitative description of the initial alignment phenomenon and a proof that the process follows a specific saddle to saddle dynamics.

Key facts. Cited by 74 papers in the map (data through 7 Oct 2026). Builds on 40 of its 59 references that are in the map. First posted on arXiv 2 Jun 2022.

Builds on

40 of 59 references.

References · 40

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2018On Lazy Training in Differentiable Programmingcited by 671
  3. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  4. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  5. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  6. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  7. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  8. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  9. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 329
  10. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
  11. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  12. 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 286
28 more
  1. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  2. 2020Loss landscapes and optimization in over-parameterized non-linear systems and neural networkscited by 231
  3. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 203
  4. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  5. 2019On the Inductive Bias of Neural Tangent Kernelscited by 177
  6. 2020Directional convergence and alignment in deep learningcited by 172
  7. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 148
  8. 2019A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Casecited by 113
  9. 2018Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundationscited by 109
  10. 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 107
  11. 2018Trainability and Accuracy of Neural Networks: An Interacting Particle System Approachcited by 105
  12. 2019How do infinite width bounded norm networks look in function space?cited by 105
  13. 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 83
  14. 2021Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Biascited by 82
  15. 2020A Unifying View on Implicit Bias in Training Linear Neural Networkscited by 74
  16. 2021What Kinds of Functions do Deep Neural Networks Learn? Insights from Variational Spline Theorycited by 60
  17. 2018Gradient Descent Quantizes ReLU Network Featurescited by 57
  18. 2021A Local Convergence Theory for Mildly Over-Parameterized Two-Layer Neural Networkcited by 47
  19. 2020Implicit Regularization in ReLU Networks with the Square Losscited by 45
  20. 2020On the Convergence of Gradient Descent Training for Two-layer ReLU-networks in the Mean Field Regimecited by 38
  21. 2020The Effects of Mild Over-parameterization on the Optimization Landscape of Shallow ReLU Neural Networkscited by 33
  22. 2022Convergence of gradient descent for deep neural networkscited by 24
  23. 2021Mean-field Analysis of Piecewise Linear Solutions for Wide ReLU Networkscited by 13
  24. 2020Sparsest Piecewise-Linear Regression of One-Dimensional Datacited by 12
  25. 2022An initial alignment between neural network and target is needed for gradient descent to learncited by 12
  26. 2022On Feature Learning in Neural Networks with Global Convergence Guaranteescited by 12
  27. 2021The Convex Geometry of Backpropagation: Neural Network Gradient Flows Converge to Extreme Points of the Dual Convex Programcited by 11
  28. 2021Existence, uniqueness, and convergence rates for gradient flows in the training of artificial neural networks with ReLU activationcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 8; 2023: 17; 2024: 17; 2025: 17; 2026: 15.

74 papers.

Sort:

By area:

  1. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  2. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 84
  3. 2023How Two-Layer Neural Networks Learn, One (Giant) Step at a Timecited by 67
  4. 2022SGD with Large Step Sizes Learns Sparse Featurescited by 66
  5. 2023Saddle-to-Saddle Dynamics in Diagonal Linear Networkscited by 54
  6. 2023Transformers learn through gradual rank increasecited by 52
  7. 2022Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Datacited by 46
  8. 2023From high-dimensional & mean-field dynamics to dimensionless ODEs: A unifying approach to SGD in two-layers networkscited by 42
  9. 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 41
  10. 2023Understanding Incremental Learning of Gradient Descent: A Fine-grained Analysis of Matrix Sensingcited by 40
  11. 2022Regression as Classification: Influence of Task Formulation on Neural Network Featurescited by 29
  12. 2023Early Neuron Alignment in Two-layer ReLU Networks with Small Initializationcited by 28
38 more
  1. 2022Incremental Learning in Diagonal Linear Networkscited by 27
  2. 2023Over-Parameterization Exponentially Slows Down Gradient Descent for Learning a Single Neuroncited by 25
  3. 2024A distributional simplicity bias in the learning dynamics of transformerscited by 23
  4. 2023Implicit regularization of deep residual networks towards neural ODEscited by 20
  5. 2023Penalising the biases in norm regularisation enforces sparsitycited by 17
  6. 2023Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputscited by 16
  7. 2024Masks, Signs, And Learning Rate Rewindingcited by 15
  8. 2025Learning In-context n-grams with Transformers: Sub-n-grams Are Near-stationary Pointscited by 15
  9. 2023Escaping mediocrity: how two-layer networks learn hard generalized linear models with SGDcited by 14
  10. 2025Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architecturescited by 14
  11. 2025An overview of condensation phenomenon in deep learningcited by 13
  12. 2024Sliding down the stairs: how correlated latent variables accelerate learning with neural networkscited by 12
  13. 2024Simplicity Bias of Two-Layer Networks beyond Linearly Separable Datacited by 11
  14. 2024Simplicity bias and optimization threshold in two-layer ReLU networkscited by 11
  15. 2025Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networkscited by 11
  16. 2024Directional Convergence Near Small Initializations and Saddles in Two-Homogeneous Neural Networkscited by 10
  17. 2023Inductive biases of multi-task learning and finetuning: multiple regimes of feature reusecited by 9
  18. 2024Learning Gaussian Multi-Index Models with Gradient Flow: Time Complexity and Directional Convergencecited by 8
  19. 2025Scaling can lead to compositional generalizationcited by 7
  20. 2023Rethinking Gauss-Newton for learning over-parameterized modelscited by 6
  21. 2024Loss Landscape of Shallow ReLU-like Neural Networks: Stationary Points, Saddle Escape, and Network Embeddingcited by 6
  22. 2024Early Directional Convergence in Deep Homogeneous Neural Networks for Small Initializationscited by 6
  23. 2024Topological obstruction to the training of shallow ReLU neural networkscited by 6
  24. 2025When Diffusion Models Memorize: Inductive Biases in Probability Flow of Minimum-Norm Shallow Neural Netscited by 6
  25. 2026Topology and Geometry of the Learning Space of ReLU Networks: Connectivity and Singularitiescited by 6
  26. 2022Magnitude and Angle Dynamics in Training Single ReLU Neuronscited by 5
  27. 2024Singular-limit analysis of gradient descent with noise injectioncited by 5
  28. 2024Can Implicit Bias Imply Adversarial Robustness?cited by 5
  29. 2025Gradient Descent with Large Step Sizes: Chaos and Fractal Convergence Regioncited by 5
  30. 2024Exploring Loss Landscapes through the Lens of Spin Glass Theorycited by 4
  31. 2025Low-rank bias, weight decay, and model merging in neural networkscited by 4
  32. 2025Saddle-To-Saddle Dynamics in Deep ReLU Networks: Low-Rank Bias in the First Saddle Escapecited by 4
  33. 2023On the Effect of Initialization: The Scaling Path of 2-Layer Neural Networkscited by 3
  34. 2024When Are Bias-Free ReLU Networks Effectively Linear Networks?cited by 3
  35. 2025A mathematical theory for understanding when abstract representations emerge in neural networkscited by 3
  36. 2023The Effect of SGD Batch Size on Autoencoder Learning: Sparsity, Sharpness, and Feature Learningcited by 2
  37. 2025Convergence of Shallow ReLU Networks on Weakly Interacting Datacited by 2
  38. 2025Structured Transformations for Stable and Interpretable Neural Computationcited by 2

50 most cited of 74 All 74 (JSON)