mlsanityData to 14 Oct 2026

Benign Overfitting in Deep Neural Networks under Lazy Training

arXiv 2305.19377 ↗First posted 30 May 2023cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

This paper focuses on over-parameterized deep neural networks (DNNs) with ReLU activation functions and proves that when the data distribution is well-separated, DNNs can achieve Bayes-optimal test error for classification while obtaining (nearly) zero-training error under the lazy training regime. For this purpose, we unify three interrelated concepts of overparameterization, benign overfitting, and the Lipschitz constant of DNNs. Our results indicate that interpolating with smoother functions leads to better generalization. Furthermore, we investigate the special case where interpolating smooth ground-truth functions is performed by DNNs under the Neural Tangent Kernel (NTK) regime for generalization. Our result demonstrates that the generalization error converges to a constant order that only depends on label noise and initialization noise, which theoretically verifies benign overfitting. Our analysis provides a tight lower bound on the normalized margin under non-smooth activation functions, as well as the minimum eigenvalue of NTK under high-dimensional settings, which has its own interest in learning theory.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 45 of its 61 references that are in the map. First posted on arXiv 30 May 2023.

Builds on

45 of 61 references.

References · 45

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  4. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  5. 2018On Lazy Training in Differentiable Programmingcited by 675
  6. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  7. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
  8. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  9. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  10. 2019Benign Overfitting in Linear Regressioncited by 549
  11. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 413
  12. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
33 more
  1. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 331
  2. 2019Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networkscited by 222
  3. 2018Just Interpolate: Kernel "Ridgeless" Regression Can Generalizecited by 190
  4. 2019On the Inductive Bias of Neural Tangent Kernelscited by 177
  5. 2016Generalization Properties of Learning with Random Featurescited by 164
  6. 2020Benign overfitting in ridge regressioncited by 153
  7. 2022Benign Overfitting in Two-layer Convolutional Neural Networkscited by 91
  8. 2020Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networkscited by 87
  9. 2019How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?cited by 84
  10. 2020Finite-sample Analysis of Interpolating Linear Classifiers in the Overparameterized Regimecited by 81
  11. 2017SGD Learns Over-parameterized Networks that Provably Generalize on Linearly Separable Datacited by 79
  12. 2020Do Wider Neural Networks Really Help Adversarial Robustness?cited by 79
  13. 2020On the Similarity between the Laplace and Neural Tangent Kernelscited by 76
  14. 2020Deep Neural Tangent Kernel and Laplace Kernel Have the Same RKHScited by 74
  15. 2021Exploring Architectural Ingredients of Adversarially Robust Deep Neural Networkscited by 74
  16. 2022Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Datacited by 74
  17. 2021Benign Overfitting of Constant-Stepsize SGD for Linear Regressioncited by 71
  18. 2019Generalization Error Bounds of Gradient Descent for Learning Over-parameterized Deep ReLU Networkscited by 60
  19. 2020A law of robustness for two-layers neural networkscited by 55
  20. 2021Uniform Convergence of Interpolators: Gaussian Width, Norm Bounds, and Benign Overfittingcited by 54
  21. 2019A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamicscited by 49
  22. 2021Benign Overfitting in Multiclass Classification: All Roads Lead to Interpolationcited by 42
  23. 2020Kernel regression in high dimensions: Refined analysis beyond double descentcited by 39
  24. 2022Memorization and Optimization in Deep Neural Networks with Minimum Over-parameterizationcited by 36
  25. 2021Towards an Understanding of Benign Overfitting in Neural Networkscited by 30
  26. 2021Foolish Crowds Support Benign Overfittingcited by 24
  27. 2021The Three Stages of Learning Dynamics in High-Dimensional Kernel Methodscited by 22
  28. 2021The Interplay Between Implicit Bias and Benign Overfitting in Two-Layer Linear Networkscited by 18
  29. 2021Tight bounds for minimum l1-norm interpolation of noisy datacited by 16
  30. 2022Generalization Properties of NAS under Activation and Skip Connection Searchcited by 14
  31. 2021On the Generalization Power of Overfitted Two-Layer Neural Tangent Kernel Modelscited by 13
  32. 2021On the Role of Optimization in Double Descent: A Least Squares Studycited by 7
  33. 2022On the Generalization Power of the Overfitted Three-Layer Neural Tangent Kernel Modelcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 5; 2025: 2; 2026: 1.

10 papers.

  1. 2023Going Beyond Linear Mode Connectivity: The Layerwise Linear Feature Connectivitycited by 50
  2. 2023On the Optimization and Generalization of Multi-head Attentioncited by 37
  3. 2024The Surprising Harmfulness of Benign Overfitting for Adversarial Robustnesscited by 9
  4. 2025Rote Learning Considered Useful: Generalizing over Memorized Data in LLMscited by 5
  5. 2024Can overfitted deep neural networks in adversarial training generalize? -- An approximation viewpointcited by 1
  6. 2025Neural Posterior Estimation with Autoregressive Tiling for Detecting Objects in Astronomical Imagescited by 1
  7. 2026When Does ℓ2-Boosting Overfit Benignly? High-Dimensional Risk Asymptotics and the ℓ1 Implicit Biascited by 1
  8. 2024Class-wise Activation Unravelling the Engima of Deep Double Descentcited by 0
  9. 2024Benign overfitting in Fixed Dimension via Physics-Informed Learning with Smooth Inductive Biascited by 0
  10. 2024Benign Overfitting under Learning Rate Conditions for α Sub-exponential Inputcited by 0