mlsanityData to 7 Oct 2026

On the Generalization Power of Overfitted Two-Layer Neural Tangent Kernel Models

arXiv 2103.05243 ↗First posted 9 Mar 2021cs.LG math.ST stat.MLcited by 13

Not on the map. Why?

Abstract (arXiv)

In this paper, we study the generalization performance of min ℓ2-norm overfitting solutions for the neural tangent kernel (NTK) model of a two-layer neural network with ReLU activation that has no bias term. We show that, depending on the ground-truth function, the test error of overfitted NTK models exhibits characteristics that are different from the "double-descent" of other overparameterized linear models with simple Fourier or Gaussian features. Specifically, for a class of learnable functions, we provide a new upper bound of the generalization error that approaches a small limiting value, even when the number of neurons p approaches infinity. This limiting value further decreases with the number of training samples n. For functions outside of this class, we provide a lower bound on the generalization error that does not diminish to zero even when n and p are both large.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 20 of its 46 references that are in the map. First posted on arXiv 9 Mar 2021.

Builds on

20 of 46 references.

References · 20

  1. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  2. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  3. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  4. 2019Benign Overfitting in Linear Regressioncited by 544
  5. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  6. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  7. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  8. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
  9. 2017High-dimensional dynamics of generalization error in neural networkscited by 238
  10. 2019Two models of double descent for weak featurescited by 194
  11. 2018To understand deep learning we need to understand kernel learningcited by 186
  12. 2019Linearized two-layers neural networks in high dimensioncited by 155
8 more
  1. 2019The Convergence Rate of Neural Networks for Learned Functions of Different Frequenciescited by 134
  2. 2019Harmless interpolation of noisy data in regressioncited by 121
  3. 2019Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networkscited by 120
  4. 2020Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regimecited by 95
  5. 2020A Random Matrix Analysis of Random Fourier Features: Beyond the Gaussian Kernel, a Precise Phase Transition, and the Corresponding Double Descentcited by 69
  6. 2020Implicit Regularization of Random Feature Modelscited by 54
  7. 2019Neural tangent kernels, transportation mappings, and universal approximationcited by 30
  8. 2019Decoupling Gating from Linearitycited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 1; 2023: 6; 2024: 2; 2025: 1; 2026: 2.

13 papers.

Sort:

  1. 2023Theory on Forgetting and Generalization of Continual Learningcited by 69
  2. 2023Benign Overfitting in Deep Neural Networks under Lazy Trainingcited by 10
  3. 2022On the Generalization Power of the Overfitted Three-Layer Neural Tangent Kernel Modelcited by 6
  4. 2021On the Double Descent of Random Features Models Trained with SGDcited by 5
  5. 2023Theoretical Characterization of the Generalization Performance of Overfitted Meta-Learningcited by 4
  6. 2023Generalization Performance of Transfer Learning: Overparameterized and Underparameterized Regimescited by 4
  7. 2023Universal Scaling Laws of Absorbing Phase Transitions in Artificial Deep Neural Networkscited by 4
  8. 2024Differentially Private Random Feature Modelcited by 2
  9. 2023Changing the Kernel During Training Leads to Double Descent in Kernel Regressioncited by 1
  10. 2025Mathematical Foundations of Neural Tangents and Infinite-Width Networkscited by 1
  11. 2024Can We Theoretically Quantify the Impacts of Local Updates on the Generalization Performance of Federated Learning?cited by 0
  12. 2026PMF-CL: Pareto-Minimal-Forgetting Continual Learner for Conflicting Taskscited by 0
1 more
  1. 2026A Rigorous, Tractable Measure of Model Complexitycited by 0