# When does gradient descent with logistic loss interpolate using deep networks with smoothed ReLU activations?

arXiv: https://arxiv.org/abs/2102.04998 · HTML: https://mlsanity.com/paper/2102.04998/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Feb 2021.

## Built on by (most cited first, 9 of 9)

- [Properties of the After Kernel](https://mlsanity.com/paper/2105.10585/) (2021)
- [Overparameterization of deep ResNet: zero loss and mean-field analysis](https://mlsanity.com/paper/2105.14417/) (2021)
- [Large Stepsize Gradient Descent for Non-Homogeneous Two-Layer Networks: Margin Improvement and Fast Optimization](https://mlsanity.com/paper/2406.08654/) (2024)
- [On Feature Learning in Neural Networks with Global Convergence Guarantees](https://mlsanity.com/paper/2204.10782/) (2022)
- [On the Global Convergence of Gradient Descent for multi-layer ResNets in the mean-field regime](https://mlsanity.com/paper/2110.02926/) (2021)
- [Early Stage Convergence and Global Convergence of Training Mildly Parameterized Neural Networks](https://mlsanity.com/paper/2206.02139/) (2022)
- [Fast Convergence in Learning Two-Layer Neural Networks with Separable Data](https://mlsanity.com/paper/2305.13471/) (2023)
- [Global Convergence of SGD For Logistic Loss on Two Layer Neural Nets](https://mlsanity.com/paper/2309.09258/) (2023)
- [Generalization of Scaled Deep ResNets in the Mean-Field Regime](https://mlsanity.com/paper/2403.09889/) (2024)

## Abstract (arXiv)

We establish conditions under which gradient descent applied to fixed-width deep networks drives the logistic loss to zero, and prove bounds on the rate of convergence. Our analysis applies for smoothed approximations to the ReLU, such as Swish and the Huberized ReLU, proposed in previous applied work. We provide two sufficient conditions for convergence. The first is simply a bound on the loss at initialization. The second is a data separation condition used in prior analyses.