Support Vectors and Gradient Dynamics of Single-Neuron ReLU Networks
Not on the map. Why?
Abstract (arXiv)
Understanding implicit bias of gradient descent for generalization capability of ReLU networks has been an important research topic in machine learning research. Unfortunately, even for a single ReLU neuron trained with the square loss, it was recently shown impossible to characterize the implicit regularization in terms of a norm of model parameters (Vardi & Shamir, 2021). In order to close the gap toward understanding intriguing generalization behavior of ReLU networks, here we examine the gradient flow dynamics in the parameter space when training single-neuron ReLU networks. Specifically, we discover an implicit bias in terms of support vectors, which plays a key role in why and how ReLU networks generalize well. Moreover, we analyze gradient flows with respect to the magnitude of the norm of initialization, and show that the norm of the learned weight strictly increases through the gradient flow. Lastly, we prove the global convergence of single ReLU neuron for d = 2 case.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 27 of its 40 references that are in the map. First posted on arXiv 11 Feb 2022.
Builds on
27 of 40 references.
References · 27
- 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
- 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
- 2017Spectrally-normalized margin bounds for neural networkscited by 686
- 2019Fantastic Generalization Measures and Where to Find Themcited by 513
- 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
- 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
- 2017Implicit Regularization in Matrix Factorizationcited by 322
- 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 314
- 2018Stronger generalization bounds for deep nets via a compression approachcited by 307
- 2020Kernel and Rich Regimes in Overparametrized Modelscited by 306
- 2015Norm-Based Capacity Control in Neural Networkscited by 299
- 2017Size-Independent Sample Complexity of Neural Networkscited by 290
15 more
- 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 266
- 2019Two models of double descent for weak featurescited by 195
- 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 177
- 2019Complexity of Linear Regions in Deep Networkscited by 167
- 2019Deep ReLU Networks Have Surprisingly Few Activation Patternscited by 161
- 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 150
- 2020Implicit Regularization in Deep Learning May Not Be Explainable by Normscited by 119
- 2019Data-dependent Sample Complexity of Deep Neural Networks via Lipschitz Augmentationcited by 71
- 2021On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descentcited by 70
- 2019Deterministic PAC-Bayesian generalization bounds for deep networks via generalizing noise-resiliencecited by 51
- 2020Implicit Regularization in ReLU Networks with the Square Losscited by 45
- 2021Learning a Single Neuron with Bias Using Gradient Descentcited by 17
- 2021Traversing the Local Polytopes of ReLU Neural Networks: A Unified Approach for Network Verificationcited by 13
- 2021Unsupervised Representation Learning via Neural Activation Codingcited by 4
- 2021Regularization by Misclassification in ReLU Neural Networkscited by 1
Built on by
Not cited yet.