# On the Generalization Power of Overfitted Two-Layer Neural Tangent Kernel Models

arXiv: https://arxiv.org/abs/2103.05243 · HTML: https://mlsanity.com/paper/2103.05243/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 46 references that are in the map.
- First posted on arXiv 9 Mar 2021.

## Builds on (most cited first, 20 of 20)

- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [Gradient Descent Provably Optimizes Over-parameterized Neural Networks](https://mlsanity.com/paper/1810.02054/) (2018)
- [Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks](https://mlsanity.com/paper/1901.08584/) (2019)
- [Benign Overfitting in Linear Regression](https://mlsanity.com/paper/1906.11300/) (2019)
- [Surprises in High-Dimensional Ridgeless Least Squares Interpolation](https://mlsanity.com/paper/1903.08560/) (2019)
- [The generalization error of random features regression: Precise asymptotics and double descent curve](https://mlsanity.com/paper/1908.05355/) (2019)
- [Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers](https://mlsanity.com/paper/1811.04918/) (2018)
- [Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data](https://mlsanity.com/paper/1808.01204/) (2018)
- [High-dimensional dynamics of generalization error in neural networks](https://mlsanity.com/paper/1710.03667/) (2017)
- [Two models of double descent for weak features](https://mlsanity.com/paper/1903.07571/) (2019)
- [To understand deep learning we need to understand kernel learning](https://mlsanity.com/paper/1802.01396/) (2018)
- [Linearized two-layers neural networks in high dimension](https://mlsanity.com/paper/1904.12191/) (2019)
- [The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies](https://mlsanity.com/paper/1906.00425/) (2019)
- [Harmless interpolation of noisy data in regression](https://mlsanity.com/paper/1903.09139/) (2019)
- [Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networks](https://mlsanity.com/paper/1909.12292/) (2019)
- [Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regime](https://mlsanity.com/paper/2003.01054/) (2020)
- [A Random Matrix Analysis of Random Fourier Features: Beyond the Gaussian Kernel, a Precise Phase Transition, and the Corresponding Double Descent](https://mlsanity.com/paper/2006.05013/) (2020)
- [Implicit Regularization of Random Feature Models](https://mlsanity.com/paper/2002.08404/) (2020)
- [Neural tangent kernels, transportation mappings, and universal approximation](https://mlsanity.com/paper/1910.06956/) (2019)
- [Decoupling Gating from Linearity](https://mlsanity.com/paper/1906.05032/) (2019)

## Built on by (most cited first, 13 of 13)

- [Theory on Forgetting and Generalization of Continual Learning](https://mlsanity.com/paper/2302.05836/) (2023)
- [Benign Overfitting in Deep Neural Networks under Lazy Training](https://mlsanity.com/paper/2305.19377/) (2023)
- [On the Generalization Power of the Overfitted Three-Layer Neural Tangent Kernel Model](https://mlsanity.com/paper/2206.02047/) (2022)
- [On the Double Descent of Random Features Models Trained with SGD](https://mlsanity.com/paper/2110.06910/) (2021)
- [Theoretical Characterization of the Generalization Performance of Overfitted Meta-Learning](https://mlsanity.com/paper/2304.04312/) (2023)
- [Generalization Performance of Transfer Learning: Overparameterized and Underparameterized Regimes](https://mlsanity.com/paper/2306.04901/) (2023)
- [Universal Scaling Laws of Absorbing Phase Transitions in Artificial Deep Neural Networks](https://mlsanity.com/paper/2307.02284/) (2023)
- [Differentially Private Random Feature Model](https://mlsanity.com/paper/2412.04785/) (2024)
- [Changing the Kernel During Training Leads to Double Descent in Kernel Regression](https://mlsanity.com/paper/2311.01762/) (2023)
- [Mathematical Foundations of Neural Tangents and Infinite-Width Networks](https://mlsanity.com/paper/2512.08264/) (2025)
- [Can We Theoretically Quantify the Impacts of Local Updates on the Generalization Performance of Federated Learning?](https://mlsanity.com/paper/2409.03863/) (2024)
- [PMF-CL: Pareto-Minimal-Forgetting Continual Learner for Conflicting Tasks](https://mlsanity.com/paper/2605.19145/) (2026)
- [A Rigorous, Tractable Measure of Model Complexity](https://mlsanity.com/paper/2605.21167/) (2026)

## Abstract (arXiv)

In this paper, we study the generalization performance of min ℓ2-norm overfitting solutions for the neural tangent kernel (NTK) model of a two-layer neural network with ReLU activation that has no bias term. We show that, depending on the ground-truth function, the test error of overfitted NTK models exhibits characteristics that are different from the "double-descent" of other overparameterized linear models with simple Fourier or Gaussian features. Specifically, for a class of learnable functions, we provide a new upper bound of the generalization error that approaches a small limiting value, even when the number of neurons p approaches infinity. This limiting value further decreases with the number of training samples n. For functions outside of this class, we provide a lower bound on the generalization error that does not diminish to zero even when n and p are both large.