# A Generalized Neural Tangent Kernel Analysis for Two-layer Neural Networks

arXiv: https://arxiv.org/abs/2002.04026 · HTML: https://mlsanity.com/paper/2002.04026/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Feb 2020.

## Built on by (most cited first, 5 of 5)

- [On the generalization of learning algorithms that do not converge](https://mlsanity.com/paper/2208.07951/) (2022)
- [Wasserstein Flow Meets Replicator Dynamics: A Mean-Field Analysis of Representation Learning in Actor-Critic](https://mlsanity.com/paper/2112.13530/) (2021)
- [An Exact Kernel Equivalence for Finite Classification Models](https://mlsanity.com/paper/2308.00824/) (2023)
- [One-pass Stochastic Gradient Descent in Overparametrized Two-layer Neural Networks](https://mlsanity.com/paper/2105.00262/) (2021)
- [Demystifying the Global Convergence Puzzle of Learning Over-parameterized ReLU Nets in Very High Dimensions](https://mlsanity.com/paper/2206.03254/) (2022)

## Abstract (arXiv)

A recent breakthrough in deep learning theory shows that the training of over-parameterized deep neural networks can be characterized by a kernel function called textitneural tangent kernel (NTK). However, it is known that this type of results does not perfectly match the practice, as NTK-based analysis requires the network weights to stay very close to their initialization throughout training, and cannot handle regularizers or gradient noises. In this paper, we provide a generalized neural tangent kernel analysis and show that noisy gradient descent with weight decay can still exhibit a "kernel-like" behavior. This implies that the training loss converges linearly up to a certain accuracy. We also establish a novel generalization error bound for two-layer neural networks trained by noisy gradient descent with weight decay.