mlsanityData to 7 Oct 2026

Magnitude and Angle Dynamics in Training Single ReLU Neurons

arXiv 2209.13394 ↗First posted 27 Sep 2022cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

To understand learning the dynamics of deep ReLU networks, we investigate the dynamic system of gradient flow w(t) by decomposing it to magnitude w(t) and angle φ(t):= π - θ(t) components. In particular, for multi-layer single ReLU neurons with spherically symmetric data distribution and the square loss function, we provide upper and lower bounds for magnitude and angle components to describe the dynamics of gradient flow. Using the obtained bounds, we conclude that small scale initialization induces slow convergence speed for deep single ReLU neurons. Finally, by exploiting the relation of gradient flow and gradient descent, we extend our results to the gradient descent approach. All theoretical results are verified by experiments.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 38 of its 47 references that are in the map. First posted on arXiv 27 Sep 2022.

Builds on

38 of 47 references.

References · 38

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  3. 2018On Lazy Training in Differentiable Programmingcited by 671
  4. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  5. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  6. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  7. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 494
  8. 2017Implicit Regularization in Matrix Factorizationcited by 318
  9. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
  10. 2019Two models of double descent for weak featurescited by 194
  11. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  12. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 172
26 more
  1. 2019Complexity of Linear Regions in Deep Networkscited by 166
  2. 2019Deep ReLU Networks Have Surprisingly Few Activation Patternscited by 160
  3. 2018Risk and parameter convergence of logistic regressioncited by 81
  4. 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 74
  5. 2020Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networkscited by 73
  6. 2020Implicit Bias in Deep Linear Classification: Initialization Scale vs Training Accuracycited by 70
  7. 2020Global Convergence of Deep Networks with One Wide Layer Followed by Pyramidal Topologycited by 66
  8. 2017Globally Optimal Gradient Descent for a ConvNet with Gaussian Inputscited by 61
  9. 2018Gradient Descent Quantizes ReLU Network Featurescited by 57
  10. 2017An Analytical Formula of Population Gradient for two-layered ReLU network and its Applications in Convergence and Critical Point Analysiscited by 52
  11. 2018Learning One-hidden-layer ReLU Networks via Gradient Descentcited by 50
  12. 2020Learning a Single Neuron with Gradient Methodscited by 50
  13. 2020Agnostic Learning of a Single Neuron with Gradient Descentcited by 46
  14. 2020Implicit Regularization in ReLU Networks with the Square Losscited by 45
  15. 2020On the Global Convergence of Training Deep Linear ResNetscited by 29
  16. 2022Hardness of Noise-Free Learning for Two-Hidden-Layer Neural Networkscited by 27
  17. 2018Exponential Convergence Time of Gradient Descent for One-Dimensional Deep Linear Neural Networkscited by 22
  18. 2017When is a Convolutional Filter Easy To Learn?cited by 21
  19. 2022Global Convergence of Sub-gradient Method for Robust Matrix Recovery: Small Initialization, Noisy Measurements, and Over-parameterizationcited by 21
  20. 2022Learning a Single Neuron with Adversarial Label Noise via Gradient Descentcited by 20
  21. 2021Learning a Single Neuron with Bias Using Gradient Descentcited by 17
  22. 2021A global convergence theory for deep ReLU implicit networks via over-parameterizationcited by 17
  23. 2021Analytic Study of Families of Spurious Minima in Two-Layer ReLU Neural Networks: A Tale of Symmetry IIcited by 16
  24. 2022Learning a Single Neuron for Non-monotonic Activation Functionscited by 13
  25. 2019Fitting ReLUs via SGD and Quantized SGDcited by 10
  26. 2021A Contraction Theory Approach to Optimization Algorithms from Acceleration Flowscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 3; 2025: 0; 2026: 1.

5 papers.

  1. 2023Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputscited by 16
  2. 2024Masks, Signs, And Learning Rate Rewindingcited by 15
  3. 2024Don't Fear Peculiar Activation Functions: EUAF and Beyondcited by 7
  4. 2024Loss Landscape of Shallow ReLU-like Neural Networks: Stationary Points, Saddle Escape, and Network Embeddingcited by 6
  5. 2026Teacher Geometry Shapes Learnability in Teacher-Student Networkscited by 0