When does gradient descent with logistic loss interpolate using deep networks with smoothed ReLU activations?
Not on the map. Why?
Abstract (arXiv)
We establish conditions under which gradient descent applied to fixed-width deep networks drives the logistic loss to zero, and prove bounds on the rate of convergence. Our analysis applies for smoothed approximations to the ReLU, such as Swish and the Huberized ReLU, proposed in previous applied work. We provide two sufficient conditions for convergence. The first is simply a bound on the loss at initialization. The second is a data separation condition used in prior analyses.
Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). First posted on arXiv 9 Feb 2021.
Builds on
0 of 0 references.
No references here yet.
Built on by
Citing papers per year: 2021: 3; 2022: 2; 2023: 2; 2024: 2.
9 papers.
- 2021Properties of the After Kernelcited by 27
- 2021Overparameterization of deep ResNet: zero loss and mean-field analysiscited by 24
- 2024Large Stepsize Gradient Descent for Non-Homogeneous Two-Layer Networks: Margin Improvement and Fast Optimizationcited by 17
- 2022On Feature Learning in Neural Networks with Global Convergence Guaranteescited by 12
- 2021On the Global Convergence of Gradient Descent for multi-layer ResNets in the mean-field regimecited by 10
- 2022Early Stage Convergence and Global Convergence of Training Mildly Parameterized Neural Networkscited by 5
- 2023Fast Convergence in Learning Two-Layer Neural Networks with Separable Datacited by 3
- 2023Global Convergence of SGD For Logistic Loss on Two Layer Neural Netscited by 3
- 2024Generalization of Scaled Deep ResNets in the Mean-Field Regimecited by 3
- 2024Large Stepsize Gradient Descent for Non-Homogeneous Two-Layer Networks: Margin Improvement and Fast Optimization12 Jun 2024
- 2024Generalization of Scaled Deep ResNets in the Mean-Field Regime14 Mar 2024
- 2023Global Convergence of SGD For Logistic Loss on Two Layer Neural Nets17 Sep 2023
- 2023Fast Convergence in Learning Two-Layer Neural Networks with Separable Data22 May 2023
- 2022Early Stage Convergence and Global Convergence of Training Mildly Parameterized Neural Networks5 Jun 2022
- 2022On Feature Learning in Neural Networks with Global Convergence Guarantees22 Apr 2022
- 2021On the Global Convergence of Gradient Descent for multi-layer ResNets in the mean-field regime6 Oct 2021
- 2021Overparameterization of deep ResNet: zero loss and mean-field analysis30 May 2021
- 2021Properties of the After Kernel21 May 2021