mlsanityData to 14 Oct 2026

Directional Convergence Analysis under Spherically Symmetric Distribution

arXiv 2105.03879 ↗First posted 9 May 2021cs.LG stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

We consider the fundamental problem of learning linear predictors (i.e., separable datasets with zero margin) using neural networks with gradient flow or gradient descent. Under the assumption of spherically symmetric data distribution, we show directional convergence guarantees with exact convergence rate for two-layer non-linear networks with only two hidden nodes, and (deep) linear networks. Moreover, our discovery is built on dynamic from the initialization without both initial loss and perfect classification constraint in contrast to previous works. We also point out and study the challenges in further strengthening and generalizing our results.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 22 of its 25 references that are in the map. First posted on arXiv 9 May 2021.

Builds on

22 of 25 references.

References · 22

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  3. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  4. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
  5. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  6. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
  7. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 284
  8. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 268
  9. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 261
  10. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  11. 2019Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networkscited by 222
  12. 2018Gradient descent aligns the layers of deep linear networkscited by 182
10 more
  1. 2020Directional convergence and alignment in deep learningcited by 178
  2. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 141
  3. 2018Convergence of Gradient Descent on Separable Datacited by 110
  4. 2018Risk and parameter convergence of logistic regressioncited by 85
  5. 2018Stochastic Gradient Descent on Separable Data: Exact Convergence with a Fixed Learning Ratecited by 64
  6. 2019Generalization Error Bounds of Gradient Descent for Learning Over-parameterized Deep ReLU Networkscited by 60
  7. 2020Gradient descent follows the regularization path for general lossescited by 59
  8. 2020Learning a Single Neuron with Gradient Methodscited by 50
  9. 2020Gradient Methods Never Overfit On Separable Datacited by 29
  10. 2019Luck Matters: Understanding Training Dynamics of Deep ReLU Networkscited by 8

Built on by

Not cited yet.