mlsanityData to 14 Oct 2026

Analysis of the expected L2 error of an over-parametrized deep neural network estimate learned by gradient descent without regularization

arXiv 2311.14609 ↗First posted 24 Nov 2023stat.ML cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Recent results show that estimates defined by over-parametrized deep neural networks learned by applying gradient descent to a regularized empirical L2 risk are universally consistent and achieve good rates of convergence. In this paper, we show that the regularization term is not necessary to obtain similar results. In the case of a suitably chosen initialization of the network, a suitable number of gradient descent steps, and a suitable step size we show that an estimate without a regularization term is universally consistent for bounded predictor variables. Additionally, we show that if the regression function is H"older smooth with H"older exponent 1/2 ≤ p ≤ 1, the L2 error converges to zero with a convergence rate of approximately n-1/(1+d). Furthermore, in case of an interaction model, where the regression function consists of a sum of H"older smooth functions with d^* components, a rate of convergence is derived which does not depend on the input dimension d.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 19 of its 32 references that are in the map. First posted on arXiv 24 Nov 2023.

Builds on

19 of 32 references.

References · 19

  1. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  2. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  3. 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 433
  4. 2021Deep learning: a statistical viewpointcited by 240
  5. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 184
  6. 2018Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionalitycited by 162
  7. 2021Acquisition of Chess Knowledge in AlphaZerocited by 123
  8. 2019On the Power and Limitations of Random Features for Understanding Neural Networkscited by 103
  9. 2018Does data interpolation contradict statistical optimality?cited by 95
  10. 2020The Interpolation Phase Transition in Neural Networks: Memorization and Generalization under Lazy Trainingcited by 84
  11. 2022Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Datacited by 74
  12. 2019Deep learning is adaptive to intrinsic dimensionality of model smoothness in anisotropic Besov spacecited by 46
7 more
  1. 2017Gradient Descent Learns One-hidden-layer CNN: Don't be Afraid of Spurious Local Minimacited by 40
  2. 2020Optimal Rates for Averaged Stochastic Gradient Descent under Neural Tangent Kernel Regimecited by 35
  3. 2019Gradient Descent Finds Global Minima for Generalizable Deep Neural Networks of Practical Sizescited by 29
  4. 2019Neural Networks Learning and Memorization with (almost) no Over-Parameterizationcited by 23
  5. 2020Network size and weights size for memorization with two-layers neural networkscited by 17
  6. 2019On the rate of convergence of fully connected very deep neural network regression estimatescited by 14
  7. 2020Memorizing Gaussians with no over-parameterizaion via gradient decent on neural networkscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 1; 2026: 1.

4 papers.

  1. 2024DRM Revisited: A Complete Error Analysiscited by 4
  2. 2024Error Analysis of Three-Layer Neural Network Trained with PGD for Deep Ritz Methodcited by 1
  3. 2025Statistically guided deep learningcited by 0
  4. 2026Minimax-Optimal Generalization Bounds for Smooth Deep Neural Networks Trained by (Stochastic) Gradient Descentcited by 0