mlsanityData to 14 Oct 2026

On the Generalization Power of the Overfitted Three-Layer Neural Tangent Kernel Model

arXiv 2206.02047 ↗First posted 4 Jun 2022cs.LG math.ST stat.MLcited by 6

Not on the map. Why?

Abstract (arXiv)

In this paper, we study the generalization performance of overparameterized 3-layer NTK models. We show that, for a specific set of ground-truth functions (which we refer to as the "learnable set"), the test error of the overfitted 3-layer NTK is upper bounded by an expression that decreases with the number of neurons of the two hidden layers. Different from 2-layer NTK where there exists only one hidden-layer, the 3-layer NTK involves interactions between two hidden-layers. Our upper bound reveals that, between the two hidden-layers, the test error descends faster with respect to the number of neurons in the second hidden-layer (the one closer to the output) than with respect to that in the first hidden-layer (the one closer to the input). We also show that the learnable set of 3-layer NTK without bias is no smaller than that of 2-layer NTK models with various choices of bias in the neurons. However, in terms of the actual generalization performance, our results suggest that 3-layer NTK is much less sensitive to the choices of bias than 2-layer NTK, especially when the input dimension is large.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 12 of its 39 references that are in the map. First posted on arXiv 4 Jun 2022.

Builds on

12 of 39 references.

References · 12

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  3. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 513
  4. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 413
  5. 2019Two models of double descent for weak featurescited by 195
  6. 2019Linearized two-layers neural networks in high dimensioncited by 155
  7. 2019Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networkscited by 121
  8. 2020Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regimecited by 95
  9. 2020Deep Neural Tangent Kernel and Laplace Kernel Have the Same RKHScited by 74
  10. 2019Decoupling Gating from Linearitycited by 13
  11. 2021On the Generalization Power of Overfitted Two-Layer Neural Tangent Kernel Modelscited by 13
  12. 2021The Dynamics of Gradient Descent for Overparametrized Neural Networkscited by 11

Built on by

Cited by year

Citing papers per year: 2023: 5; 2024: 0; 2025: 1.

6 papers.

  1. 2023Theory on Forgetting and Generalization of Continual Learningcited by 72
  2. 2023Benign Overfitting in Deep Neural Networks under Lazy Trainingcited by 10
  3. 2023Generalization Guarantees of Gradient Descent for Multi-Layer Neural Networkscited by 9
  4. 2023Theoretical Characterization of the Generalization Performance of Overfitted Meta-Learningcited by 4
  5. 2023Generalization Performance of Transfer Learning: Overparameterized and Underparameterized Regimescited by 4
  6. 2025Mathematical Foundations of Neural Tangents and Infinite-Width Networkscited by 1