mlsanityData to 7 Oct 2026

Generalization Ability of Wide Neural Networks on ℝ

arXiv 2302.05933 ↗First posted 12 Feb 2023stat.ML cs.LGcited by 28

Not on the map. Why?

Abstract (arXiv)

We perform a study on the generalization ability of the wide two-layer ReLU neural network on ℝ. We first establish some spectral properties of the neural tangent kernel (NTK): a) Kd, the NTK defined on ℝd, is positive definite; b) λi(K1), the i-th largest eigenvalue of K1, is proportional to i-2. We then show that: i) when the width mrightarrow∞, the neural network kernel (NNK) uniformly converges to the NTK; ii) the minimax rate of regression over the RKHS associated to K1 is n-2/3; iii) if one adopts the early stopping strategy in training a wide neural network, the resulting neural network achieves the minimax rate; iv) if one trains the neural network till it overfits the data, the resulting neural network can not generalize well. Finally, we provide an explanation to reconcile our theory and the widely observed ``benign overfitting phenomenon''.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 47 of its 69 references that are in the map. First posted on arXiv 12 Feb 2023.

Builds on

47 of 69 references.

References · 47

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  4. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  5. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  6. 2018On Lazy Training in Differentiable Programmingcited by 671
  7. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  8. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  9. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  10. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  11. 2016Error bounds for approximations with deep ReLU networkscited by 567
  12. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
35 more
  1. 2019Benign Overfitting in Linear Regressioncited by 544
  2. 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 431
  3. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  4. 2018Just Interpolate: Kernel "Ridgeless" Regression Can Generalizecited by 190
  5. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  6. 2018To understand deep learning we need to understand kernel learningcited by 186
  7. 2019On the Inductive Bias of Neural Tangent Kernelscited by 177
  8. 2017Optimal approximation of piecewise smooth functions using deep ReLU neural networkscited by 176
  9. 2019Towards Understanding the Spectral Bias of Deep Learningcited by 173
  10. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  11. 2018Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionalitycited by 160
  12. 2021Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolationcited by 130
  13. 2018Does data interpolation contradict statistical optimality?cited by 95
  14. 2017Recovery Guarantees for One-hidden-layer Neural Networkscited by 94
  15. 2018Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approachcited by 89
  16. 2013Early stopping and non-parametric regression: An optimal data-dependent stopping rulecited by 86
  17. 2020The Interpolation Phase Transition in Neural Networks: Memorization and Generalization under Lazy Trainingcited by 84
  18. 2020Deep Equals Shallow for ReLU Networks in Kernel Regimescited by 83
  19. 2020On the Similarity between the Laplace and Neural Tangent Kernelscited by 76
  20. 2022Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Datacited by 74
  21. 2016Optimal Rates For Regularization Of Statistical Inverse Learning Problemscited by 68
  22. 2018Optimal Rates for Spectral Algorithms with Least-Squares Regression over Hilbert Spacescited by 59
  23. 2018Deep Neural Networks Learn Non-Smooth Functions Effectivelycited by 57
  24. 2020Kernel Alignment Risk Estimator: Risk Prediction from Training Datacited by 54
  25. 2018Learning One-hidden-layer ReLU Networks via Gradient Descentcited by 50
  26. 2018Fast convergence rates of deep neural networks for classificationcited by 46
  27. 2019Deep learning is adaptive to intrinsic dimensionality of model smoothness in anisotropic Besov spacecited by 46
  28. 2018Consistency of Interpolation with Laplace Kernels is a High-Dimensional Phenomenoncited by 44
  29. 2019Mean Field Analysis of Deep Neural Networkscited by 42
  30. 2020Regularization Matters: A Nonparametric Perspective on Overparametrized Neural Networkcited by 38
  31. 2022Benign, Tempered, or Catastrophic: A Taxonomy of Overfittingcited by 36
  32. 2022Limitations of the NTK for Understanding Generalization in Deep Learningcited by 34
  33. 2019On the minimax optimality and superiority of deep neural network learning over sparse parameter spacescited by 24
  34. 2022Stability and Generalization Analysis of Gradient Methods for Shallow Neural Networkscited by 21
  35. 2021Non-asymptotic approximations of neural networks by Gaussian processescited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 9; 2025: 7; 2026: 3.

28 papers.

Sort:

  1. 2023On the Eigenvalue Decay Rates of a Class of Neural-Network Related Kernel Functions Defined on General Domainscited by 30
  2. 2023Mind the spikes: Benign overfitting of kernels and neural networks in fixed dimensioncited by 26
  3. 2024A Survey on Statistical Theory of Deep Learning: Approximation, Training Dynamics, and Generative Modelscited by 20
  4. 2023On the Asymptotic Learning Curves of Kernel Ridge Regression under Power-law Decaycited by 19
  5. 2023Kernel interpolation generalizes poorlycited by 16
  6. 2024The Positivity of the Neural Tangent Kernelcited by 12
  7. 2023Noisy Interpolation Learning with Shallow Univariate ReLU Networkscited by 11
  8. 2023Optimal Rate of Kernel Regression in Large Dimensionscited by 10
  9. 2023How many Neurons do we need? A refined Analysis for Shallow Networks trained with Gradient Descentcited by 10
  10. 2023Marginal Density Ratio for Off-Policy Evaluation in Contextual Banditscited by 9
  11. 2024Optimal Rates of Kernel Ridge Regression under Source Condition in Large Dimensionscited by 9
  12. 2024The phase diagram of kernel interpolation in large dimensionscited by 9
16 more
  1. 2023Generalization Ability of Wide Residual Networkscited by 5
  2. 2024On the Impacts of the Random Initialization in the Neural Tangent Kernel Theorycited by 5
  3. 2024Provable Tempered Overfitting of Minimal Nets and Typical Netscited by 5
  4. 2025Determinant Estimation under Memory Constraints and Neural Scaling Lawscited by 3
  5. 2025Neural Tangent Kernel of Neural Networks with Loss Informed by Differential Operatorscited by 3
  6. 2025Divergence of Empirical Neural Tangent Kernel in Classification Problemscited by 3
  7. 2025Beyond ReLU: How Activations Affect Neural Kernels and Random Wide Networkscited by 3
  8. 2025On the Saturation Effects of Spectral Algorithms in Large Dimensionscited by 2
  9. 2024The Optimality of Kernel Classifiers in Sobolev Spacecited by 1
  10. 2024On the Pinsker bound of inner product kernel regression in large dimensionscited by 1
  11. 2024Towards a Statistical Understanding of Neural Networks: Beyond the Neural Tangent Kernel Theoriescited by 0
  12. 2025Variational Deep Learning via Implicit Regularizationcited by 0
  13. 2025A ZeNN architecture to avoid the Gaussian trapcited by 0
  14. 2026Adversarial Robustness of NTK Neural Networkscited by 0
  15. 2026Large Dimensional Kernel Ridge Regression: Extending to Product Kernelscited by 0
  16. 2026Optimal Rates for Generalization of Gradient Descent Methods with Deep Neural Networkscited by 0