# Generalization Ability of Wide Neural Networks on ℝ

arXiv: https://arxiv.org/abs/2302.05933 · HTML: https://mlsanity.com/paper/2302.05933/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 69 references that are in the map.
- First posted on arXiv 12 Feb 2023.

## Builds on (most cited first, 30 of 47)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [A Convergence Theory for Deep Learning via Over-Parameterization](https://mlsanity.com/paper/1811.03962/) (2018)
- [Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent](https://mlsanity.com/paper/1902.06720/) (2019)
- [On Lazy Training in Differentiable Programming](https://mlsanity.com/paper/1812.07956/) (2018)
- [Deep Double Descent: Where Bigger Models and More Data Hurt](https://mlsanity.com/paper/1912.02292/) (2019)
- [Gradient Descent Provably Optimizes Over-parameterized Neural Networks](https://mlsanity.com/paper/1810.02054/) (2018)
- [Error bounds for approximations with deep ReLU networks](https://mlsanity.com/paper/1610.01145/) (2016)
- [Gradient Descent Finds Global Minima of Deep Neural Networks](https://mlsanity.com/paper/1811.03804/) (2018)
- [On Exact Computation with an Infinitely Wide Neural Net](https://mlsanity.com/paper/1904.11955/) (2019)
- [Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks](https://mlsanity.com/paper/1901.08584/) (2019)
- [Benign Overfitting in Linear Regression](https://mlsanity.com/paper/1906.11300/) (2019)
- [Nonparametric regression using deep neural networks with ReLU activation function](https://mlsanity.com/paper/1708.06633/) (2017)
- [Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks](https://mlsanity.com/paper/2002.02561/) (2020)
- [Just Interpolate: Kernel "Ridgeless" Regression Can Generalize](https://mlsanity.com/paper/1808.00387/) (2018)
- [To understand deep learning we need to understand kernel learning](https://mlsanity.com/paper/1802.01396/) (2018)
- [Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit](https://mlsanity.com/paper/1902.06015/) (2019)
- [Optimal approximation of piecewise smooth functions using deep ReLU neural networks](https://mlsanity.com/paper/1709.05289/) (2017)
- [On the Inductive Bias of Neural Tangent Kernels](https://mlsanity.com/paper/1905.12173/) (2019)
- [Towards Understanding the Spectral Bias of Deep Learning](https://mlsanity.com/paper/1912.01198/) (2019)
- [Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networks](https://mlsanity.com/paper/2006.13198/) (2020)
- [Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality](https://mlsanity.com/paper/1810.08033/) (2018)
- [Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation](https://mlsanity.com/paper/2105.14368/) (2021)
- [Does data interpolation contradict statistical optimality?](https://mlsanity.com/paper/1806.09471/) (2018)
- [Recovery Guarantees for One-hidden-layer Neural Networks](https://mlsanity.com/paper/1706.03175/) (2017)
- [Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach](https://mlsanity.com/paper/1806.01316/) (2018)
- [Early stopping and non-parametric regression: An optimal data-dependent stopping rule](https://mlsanity.com/paper/1306.3574/) (2013)
- [Deep Equals Shallow for ReLU Networks in Kernel Regimes](https://mlsanity.com/paper/2009.14397/) (2020)
- [The Interpolation Phase Transition in Neural Networks: Memorization and Generalization under Lazy Training](https://mlsanity.com/paper/2007.12826/) (2020)

## Built on by (most cited first, 30 of 30)

- [On the Eigenvalue Decay Rates of a Class of Neural-Network Related Kernel Functions Defined on General Domains](https://mlsanity.com/paper/2305.02657/) (2023)
- [Mind the spikes: Benign overfitting of kernels and neural networks in fixed dimension](https://mlsanity.com/paper/2305.14077/) (2023)
- [A Survey on Statistical Theory of Deep Learning: Approximation, Training Dynamics, and Generative Models](https://mlsanity.com/paper/2401.07187/) (2024)
- [On the Asymptotic Learning Curves of Kernel Ridge Regression under Power-law Decay](https://mlsanity.com/paper/2309.13337/) (2023)
- [Kernel interpolation generalizes poorly](https://mlsanity.com/paper/2303.15809/) (2023)
- [The Positivity of the Neural Tangent Kernel](https://mlsanity.com/paper/2404.12928/) (2024)
- [Noisy Interpolation Learning with Shallow Univariate ReLU Networks](https://mlsanity.com/paper/2307.15396/) (2023)
- [Optimal Rate of Kernel Regression in Large Dimensions](https://mlsanity.com/paper/2309.04268/) (2023)
- [Optimal Rates of Kernel Ridge Regression under Source Condition in Large Dimensions](https://mlsanity.com/paper/2401.01270/) (2024)
- [How many Neurons do we need? A refined Analysis for Shallow Networks trained with Gradient Descent](https://mlsanity.com/paper/2309.08044/) (2023)
- [Marginal Density Ratio for Off-Policy Evaluation in Contextual Bandits](https://mlsanity.com/paper/2312.01457/) (2023)
- [The phase diagram of kernel interpolation in large dimensions](https://mlsanity.com/paper/2404.12597/) (2024)
- [Generalization Ability of Wide Residual Networks](https://mlsanity.com/paper/2305.18506/) (2023)
- [On the Impacts of the Random Initialization in the Neural Tangent Kernel Theory](https://mlsanity.com/paper/2410.05626/) (2024)
- [Provable Tempered Overfitting of Minimal Nets and Typical Nets](https://mlsanity.com/paper/2410.19092/) (2024)
- [Determinant Estimation under Memory Constraints and Neural Scaling Laws](https://mlsanity.com/paper/2503.04424/) (2025)
- [Neural Tangent Kernel of Neural Networks with Loss Informed by Differential Operators](https://mlsanity.com/paper/2503.11029/) (2025)
- [Divergence of Empirical Neural Tangent Kernel in Classification Problems](https://mlsanity.com/paper/2504.11130/) (2025)
- [Beyond ReLU: How Activations Affect Neural Kernels and Random Wide Networks](https://mlsanity.com/paper/2506.22429/) (2025)
- [On the Pinsker bound of inner product kernel regression in large dimensions](https://mlsanity.com/paper/2409.00915/) (2024)
- [On the Saturation Effects of Spectral Algorithms in Large Dimensions](https://mlsanity.com/paper/2503.00504/) (2025)
- [Implicit Bias of SGD in Multivariate ReLU Networks: Effective Width Collapse](https://mlsanity.com/paper/2607.03613/) (2026)
- [The Optimality of Kernel Classifiers in Sobolev Space](https://mlsanity.com/paper/2402.01148/) (2024)
- [Optimal Confidence Band for Kernel Gradient Flow Estimator](https://mlsanity.com/paper/2605.05768/) (2026)
- [Towards a Statistical Understanding of Neural Networks: Beyond the Neural Tangent Kernel Theories](https://mlsanity.com/paper/2412.18756/) (2024)
- [Variational Deep Learning via Implicit Regularization](https://mlsanity.com/paper/2505.20235/) (2025)
- [A ZeNN architecture to avoid the Gaussian trap](https://mlsanity.com/paper/2505.20553/) (2025)
- [Adversarial Robustness of NTK Neural Networks](https://mlsanity.com/paper/2604.25965/) (2026)
- [Large Dimensional Kernel Ridge Regression: Extending to Product Kernels](https://mlsanity.com/paper/2605.14524/) (2026)
- [Optimal Rates for Generalization of Gradient Descent Methods with Deep Neural Networks](https://mlsanity.com/paper/2606.06764/) (2026)

## Abstract (arXiv)

We perform a study on the generalization ability of the wide two-layer ReLU neural network on ℝ. We first establish some spectral properties of the neural tangent kernel (NTK): a) Kd, the NTK defined on ℝd, is positive definite; b) λi(K1), the i-th largest eigenvalue of K1, is proportional to i-2. We then show that: i) when the width mrightarrow∞, the neural network kernel (NNK) uniformly converges to the NTK; ii) the minimax rate of regression over the RKHS associated to K1 is n-2/3; iii) if one adopts the early stopping strategy in training a wide neural network, the resulting neural network achieves the minimax rate; iv) if one trains the neural network till it overfits the data, the resulting neural network can not generalize well. Finally, we provide an explanation to reconcile our theory and the widely observed ``benign overfitting phenomenon''.