mlsanityData to 7 Oct 2026

How many Neurons do we need? A refined Analysis for Shallow Networks trained with Gradient Descent

arXiv 2309.08044 ↗First posted 14 Sep 2023stat.ML cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

We analyze the generalization properties of two-layer neural networks in the neural tangent kernel (NTK) regime, trained with gradient descent (GD). For early stopped GD we derive fast rates of convergence that are known to be minimax optimal in the framework of non-parametric regression in reproducing kernel Hilbert spaces. On our way, we precisely keep track of the number of hidden neurons required for generalization and improve over existing results. We further show that the weights during training remain in a vicinity around initialization, the radius being dependent on structural assumptions such as degree of smoothness of the regression function and eigenvalue decay of the integral operator associated to the NTK.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 31 of its 40 references that are in the map. First posted on arXiv 14 Sep 2023.

Builds on

31 of 40 references.

References · 31

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  3. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  4. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  5. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  6. 2019Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networkscited by 221
  7. 2018To understand deep learning we need to understand kernel learningcited by 186
  8. 2019On the Inductive Bias of Neural Tangent Kernelscited by 177
  9. 2016Generalization Properties of Learning with Random Featurescited by 163
  10. 2016A vector-contraction inequality for Rademacher complexitiescited by 141
  11. 2018Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networkscited by 140
  12. 2016Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivitycited by 138
19 more
  1. 2018Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernelcited by 131
  2. 2020Deep Equals Shallow for ReLU Networks in Kernel Regimescited by 83
  3. 2018On the Convergence Rate of Training Recurrent Neural Networkscited by 82
  4. 2020Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networkscited by 76
  5. 2020On the Similarity between the Laplace and Neural Tangent Kernelscited by 76
  6. 2020Deep Neural Tangent Kernel and Laplace Kernel Have the Same RKHScited by 74
  7. 2016Optimal Rates For Regularization Of Statistical Inverse Learning Problemscited by 68
  8. 2019Quadratic Suffices for Over-parametrization via Matrix Chernoff Boundcited by 58
  9. 2019A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamicscited by 49
  10. 2018Uniform Convergence of Gradients for Non-Convex Learning and Optimizationcited by 38
  11. 2020Optimal Rates for Averaged Stochastic Gradient Descent under Neural Tangent Kernel Regimecited by 35
  12. 2018Learning with SGD and Random Featurescited by 33
  13. 2019A type of generalization error induced by initialization in deep neural networkscited by 33
  14. 2023Generalization Ability of Wide Neural Networks on ℝcited by 28
  15. 2022Neural Tangent Kernel: A Surveycited by 22
  16. 2019Implicit Regularization of Accelerated Methods in Hilbert Spacescited by 14
  17. 2021On the existence of global minima and convergence analyses for gradient descent methods in the training of deep neural networkscited by 12
  18. 2022Neural Tangent Kernel Analysis of Deep Narrow Neural Networkscited by 11
  19. 2021From inexact optimization to learning via gradient concentrationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 5.

10 papers.

  1. 2024Optimal Convergence Rates for Neural Operatorscited by 7
  2. 2024Generalization Analysis of Online Stochastic Gradient Descent for Overparameterized Two-Layer Neural Networkscited by 4
  3. 2025Neuroplasticity in Artificial Intelligence -- An Overview and Inspirations on Drop In & Out Learningcited by 3
  4. 2025Random feature approximation for general spectral methodscited by 2
  5. 2026Optimization, Generalization and Differential Privacy Bounds for Gradient Descent on Kolmogorov-Arnold Networkscited by 1
  6. 2025Finite Samples for Shallow Neural Networkscited by 0
  7. 2026Random Features for Operator-Valued Kernels: Bridging Kernel Methods and Neural Operatorscited by 0
  8. 2026Optimal Rates for Generalization of Gradient Descent Methods with Deep Neural Networkscited by 0
  9. 2026Minimax-Optimal Generalization Bounds for Smooth Deep Neural Networks Trained by (Stochastic) Gradient Descentcited by 0
  10. 2026A Smoothed Discrepancy Principle for Random Feature Methods and Neural Networkscited by 0