mlsanityData to 7 Oct 2026

Regularization Matters: A Nonparametric Perspective on Overparametrized Neural Network

arXiv 2007.02486 ↗First posted 6 Jul 2020stat.ML cs.LGcited by 38

Not on the map. Why?

Abstract (arXiv)

Overparametrized neural networks trained by gradient descent (GD) can provably overfit any training data. However, the generalization guarantee may not hold for noisy data. From a nonparametric perspective, this paper studies how well overparametrized neural networks can recover the true target function in the presence of random noises. We establish a lower bound on the L2 estimation error with respect to the GD iterations, which is away from zero without a delicate scheme of early stopping. In turn, through a comprehensive analysis of ℓ2-regularized GD trajectories, we prove that for overparametrized one-hidden-layer ReLU neural network with the ℓ2 regularization: (1) the output is close to that of the kernel ridge regression with the corresponding neural tangent kernel; (2) minimax optimal rate of L2 estimation error can be achieved. Numerical experiments confirm our theory and further demonstrate that the ℓ2 regularization approach improves the training robustness and works for a wider range of neural networks.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). First posted on arXiv 6 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 5; 2023: 11; 2024: 6; 2025: 6; 2026: 4.

38 papers.

Sort:

By area:

  1. 2022Neural Networks Efficiently Learn Low-Dimensional Representations with SGDcited by 61
  2. 2024Neural Network-Based Score Estimation in Diffusion Models: Optimization and Generalizationcited by 39
  3. 2023On the Eigenvalue Decay Rates of a Class of Neural-Network Related Kernel Functions Defined on General Domainscited by 30
  4. 2023Generalization Ability of Wide Neural Networks on ℝcited by 28
  5. 2023Mind the spikes: Benign overfitting of kernels and neural networks in fixed dimensioncited by 26
  6. 2021Stability & Generalisation of Gradient Descent for Shallow Neural Networks without the Neural Tangent Kernelcited by 25
  7. 2021Uniform Generalization Bounds for Overparameterized Neural Networkscited by 21
  8. 2022Stability and Generalization Analysis of Gradient Methods for Shallow Neural Networkscited by 21
  9. 2024A Survey on Statistical Theory of Deep Learning: Approximation, Training Dynamics, and Generative Modelscited by 20
  10. 2023Complexity Matters: Rethinking the Latent Space for Generative Modelingcited by 19
  11. 2021On the Equivalence between Neural Network and Support Vector Machinecited by 17
  12. 2021Understanding Square Loss in Training Overparametrized Neural Network Classifierscited by 14
26 more
  1. 2023Post-hoc Bias Scoring Is Optimal For Fair Classificationcited by 14
  2. 2022Sobolev Acceleration and Statistical Optimality for Learning Elliptic Equations via Gradient Descentcited by 11
  3. 2023Exact Count of Boundary Pieces of ReLU Classifiers: Towards the Proper Complexity Measure for Classificationcited by 11
  4. 2023Optimal Rate of Kernel Regression in Large Dimensionscited by 10
  5. 2023Random Smoothing Regularization in Kernel Gradient Descent Learningcited by 6
  6. 2021Nonparametric Regression with Shallow Overparameterized Neural Networks Trained by GD with Early Stoppingcited by 5
  7. 2022Learning Lipschitz Functions by GD-trained Shallow Overparameterized ReLU Neural Networkscited by 5
  8. 2024On the Impacts of the Random Initialization in the Neural Tangent Kernel Theorycited by 5
  9. 2022Unlabeled Data Help: Minimax Analysis and Adversarial Robustnesscited by 4
  10. 2025Low-rank bias, weight decay, and model merging in neural networkscited by 4
  11. 2025Neural Tangent Kernel of Neural Networks with Loss Informed by Differential Operatorscited by 3
  12. 2025Divergence of Empirical Neural Tangent Kernel in Classification Problemscited by 3
  13. 2021Generalization of GANs and overparameterized models under Lipschitz continuitycited by 2
  14. 2024Parameter-Efficient Fine-Tuning for Continual Learning: A Neural Tangent Kernel Perspectivecited by 2
  15. 2024Gradient Descent Finds Over-Parameterized Neural Networks with Sharp Generalization for Nonparametric Regressioncited by 2
  16. 2025On the Saturation Effects of Spectral Algorithms in Large Dimensionscited by 2
  17. 2023Mixture Weight Estimation and Model Prediction in Multi-source Multi-target Domain Adaptationcited by 1
  18. 2023Bootstrap Your Own Variancecited by 1
  19. 2024How scanning probe microscopy can be supported by Artificial Intelligence and quantum computingcited by 1
  20. 2025Sampling Complexity of TD and PPO in RKHScited by 1
  21. 2023Random Linear Projections Loss for Hyperplane-Based Optimization in Neural Networkscited by 0
  22. 2025Shallow Neural Networks Learn Low-Degree Spherical Polynomials with Feature Learning by Learnable Channel Attentioncited by 0
  23. 2026Gradient Descent with Projection Finds Over-Parameterized Neural Networks for Learning Low-Degree Polynomials with Nearly Minimax Optimal Ratecited by 0
  24. 2026Exact Certification of Neural Networks and Partition Aggregation Ensembles against Label Poisoningcited by 0
  25. 2026Optimal ridge regularization revisitedcited by 0
  26. 2026Optimal Rates for Generalization of Gradient Descent Methods with Deep Neural Networkscited by 0