# Analysis of the expected L2 error of an over-parametrized deep neural network estimate learned by gradient descent without regularization

arXiv: https://arxiv.org/abs/2311.14609 · HTML: https://mlsanity.com/paper/2311.14609/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 32 references that are in the map.
- First posted on arXiv 24 Nov 2023.

## Builds on (most cited first, 19 of 19)

- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [A Convergence Theory for Deep Learning via Over-Parameterization](https://mlsanity.com/paper/1811.03962/) (2018)
- [Nonparametric regression using deep neural networks with ReLU activation function](https://mlsanity.com/paper/1708.06633/) (2017)
- [Deep learning: a statistical viewpoint](https://mlsanity.com/paper/2103.09177/) (2021)
- [A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks](https://mlsanity.com/paper/1810.02281/) (2018)
- [Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality](https://mlsanity.com/paper/1810.08033/) (2018)
- [Acquisition of Chess Knowledge in AlphaZero](https://mlsanity.com/paper/2111.09259/) (2021)
- [On the Power and Limitations of Random Features for Understanding Neural Networks](https://mlsanity.com/paper/1904.00687/) (2019)
- [Does data interpolation contradict statistical optimality?](https://mlsanity.com/paper/1806.09471/) (2018)
- [The Interpolation Phase Transition in Neural Networks: Memorization and Generalization under Lazy Training](https://mlsanity.com/paper/2007.12826/) (2020)
- [Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Data](https://mlsanity.com/paper/2202.05928/) (2022)
- [Deep learning is adaptive to intrinsic dimensionality of model smoothness in anisotropic Besov space](https://mlsanity.com/paper/1910.12799/) (2019)
- [Gradient Descent Learns One-hidden-layer CNN: Don't be Afraid of Spurious Local Minima](https://mlsanity.com/paper/1712.00779/) (2017)
- [Optimal Rates for Averaged Stochastic Gradient Descent under Neural Tangent Kernel Regime](https://mlsanity.com/paper/2006.12297/) (2020)
- [Gradient Descent Finds Global Minima for Generalizable Deep Neural Networks of Practical Sizes](https://mlsanity.com/paper/1908.02419/) (2019)
- [Neural Networks Learning and Memorization with (almost) no Over-Parameterization](https://mlsanity.com/paper/1911.09873/) (2019)
- [Network size and weights size for memorization with two-layers neural networks](https://mlsanity.com/paper/2006.02855/) (2020)
- [On the rate of convergence of fully connected very deep neural network regression estimates](https://mlsanity.com/paper/1908.11133/) (2019)
- [Memorizing Gaussians with no over-parameterizaion via gradient decent on neural networks](https://mlsanity.com/paper/2003.12895/) (2020)

## Built on by (most cited first, 4 of 4)

- [DRM Revisited: A Complete Error Analysis](https://mlsanity.com/paper/2407.09032/) (2024)
- [Error Analysis of Three-Layer Neural Network Trained with PGD for Deep Ritz Method](https://mlsanity.com/paper/2405.11451/) (2024)
- [Statistically guided deep learning](https://mlsanity.com/paper/2504.08489/) (2025)
- [Minimax-Optimal Generalization Bounds for Smooth Deep Neural Networks Trained by (Stochastic) Gradient Descent](https://mlsanity.com/paper/2606.06772/) (2026)

## Abstract (arXiv)

Recent results show that estimates defined by over-parametrized deep neural networks learned by applying gradient descent to a regularized empirical L2 risk are universally consistent and achieve good rates of convergence. In this paper, we show that the regularization term is not necessary to obtain similar results. In the case of a suitably chosen initialization of the network, a suitable number of gradient descent steps, and a suitable step size we show that an estimate without a regularization term is universally consistent for bounded predictor variables. Additionally, we show that if the regression function is H"older smooth with H"older exponent 1/2 ≤ p ≤ 1, the L2 error converges to zero with a convergence rate of approximately n-1/(1+d). Furthermore, in case of an interaction model, where the regression function consists of a sum of H"older smooth functions with d^* components, a rate of convergence is derived which does not depend on the input dimension d.