The late-stage training dynamics of (stochastic) subgradient descent on homogeneous neural networks
Not on the map. Why?
Abstract (arXiv)
We analyze the implicit bias of constant step stochastic subgradient descent (SGD). We consider the setting of binary classification with homogeneous neural networks - a large class of deep neural networks with ReLU-type activation functions such as MLPs and CNNs without biases. We interpret the dynamics of normalized SGD iterates as an Euler-like discretization of a conservative field flow that is naturally associated to the normalized classification margin. Owing to this interpretation, we show that normalized SGD iterates converge to the set of critical points of the normalized margin at late-stage training (i.e., assuming that the data is correctly classified with positive normalized margin). Up to our knowledge, this is the first extension of the analysis of Lyu and Li (2020) on the discrete dynamics of gradient descent to the nonsmooth and stochastic setting. Our main result applies to binary classification with exponential or logistic losses. We additionally discuss extensions to more general settings.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 29 of its 52 references that are in the map. First posted on arXiv 8 Feb 2025.
Builds on
29 of 52 references.
References · 29
- 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
- 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
- 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 342
- 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
- 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 268
- 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 266
- 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
- 2018Gradient descent aligns the layers of deep linear networkscited by 182
- 2020Directional convergence and alignment in deep learningcited by 178
- 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 177
- 2018Stochastic subgradient method converges on tame functionscited by 124
- 2018Convergence of Gradient Descent on Separable Datacited by 110
17 more
- 2022On the Implicit Bias in Deep-Learning Algorithmscited by 86
- 2018Risk and parameter convergence of logistic regressioncited by 85
- 2020A Unifying View on Implicit Bias in Training Linear Neural Networkscited by 75
- 2019Conservative set valued fields, automatic differentiation, stochastic gradient method and deep learningcited by 72
- 2018Stochastic Gradient Descent on Separable Data: Exact Convergence with a Fixed Learning Ratecited by 64
- 2019Lexicographic and Depth-Sensitive Margins in Homogeneous and Non-Homogeneous Deep Modelscited by 53
- 2017Stochastic Methods for Composite and Weakly Convex Optimization Problemscited by 48
- 2020The Implicit Bias for Adaptive Optimization Algorithms on Homogeneous Neural Networkscited by 46
- 2021On Margin Maximization in Linear and ReLU Networkscited by 34
- 2022Training invariances and the low-rank phenomenon: beyond linear networkscited by 33
- 2022The Asymmetric Maximum Margin Bias of Quasi-Homogeneous Neural Networkscited by 25
- 2018Analysis of nonsmooth stochastic approximation: the differential inclusion approachcited by 23
- 2024Large Stepsize Gradient Descent for Non-Homogeneous Two-Layer Networks: Margin Improvement and Fast Optimizationcited by 17
- 2023Nonsmooth nonconvex stochastic heavy ballcited by 8
- 2015A Generalization of the Borkar-Meyn Theorem for Stochastic Recursive Inclusionscited by 7
- 2022Subgradient sampling for nonsmooth nonconvex minimizationcited by 6
- 2023Global stability of first-order methods for coercive tame functionscited by 6
Built on by
Citing papers per year: 2025: 1.
1 paper.