# Neural Tangent Kernel Analysis of Deep Narrow Neural Networks

arXiv: https://arxiv.org/abs/2202.02981 · HTML: https://mlsanity.com/paper/2202.02981/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 74 references that are in the map.
- First posted on arXiv 7 Feb 2022.

## Builds on (most cited first, 11 of 11)

- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [Highway Networks](https://mlsanity.com/paper/1505.00387/) (2015)
- [Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation](https://mlsanity.com/paper/1902.04760/) (2019)
- [Approximating Continuous Functions by ReLU Nets of Minimal Width](https://mlsanity.com/paper/1710.11278/) (2017)
- [A Rigorous Framework for the Mean Field Limit of Multilayer Neural Networks](https://mlsanity.com/paper/2001.11443/) (2020)
- [On the infinite width limit of neural networks with a standard parameterization](https://mlsanity.com/paper/2001.07301/) (2020)
- [Overparameterization of deep ResNet: zero loss and mean-field analysis](https://mlsanity.com/paper/2105.14417/) (2021)
- [On the Global Convergence of Gradient Descent for multi-layer ResNets in the mean-field regime](https://mlsanity.com/paper/2110.02926/) (2021)
- [Disentangling feature and lazy training in deep neural networks](https://mlsanity.com/paper/1906.08034/) (2019)

## Built on by (most cited first, 11 of 11)

- [How many Neurons do we need? A refined Analysis for Shallow Networks trained with Gradient Descent](https://mlsanity.com/paper/2309.08044/) (2023)
- [Information-Theoretic Generalization Bounds for Deep Neural Networks](https://mlsanity.com/paper/2404.03176/) (2024)
- [Theoretical Analysis of Robust Overfitting for Wide DNNs: An NTK Approach](https://mlsanity.com/paper/2310.06112/) (2023)
- [Approximation results for Gradient Descent trained Shallow Neural Networks in 1d](https://mlsanity.com/paper/2209.08399/) (2022)
- [Approximation Results for Gradient Descent trained Neural Networks](https://mlsanity.com/paper/2309.04860/) (2023)
- [Non-vacuous Generalization Bounds for Deep Neural Networks without any modification to the trained models](https://mlsanity.com/paper/2503.07325/) (2025)
- [Approximation and Gradient Descent Training with Neural Networks](https://mlsanity.com/paper/2405.11696/) (2024)
- [Depth-induced NTK: Bridging Over-parameterized Neural Networks and Deep Neural Kernels](https://mlsanity.com/paper/2511.05585/) (2025)
- [Compact Vision Transformer by Reduction of Kernel Complexity](https://mlsanity.com/paper/2507.12780/) (2025)
- [Revisiting the Neural Tangent Kernel: the role of large width and depth](https://mlsanity.com/paper/2511.07272/) (2025)
- [A Gaussian Comparison Theorem for Training Dynamics in Machine Learning](https://mlsanity.com/paper/2603.09310/) (2026)

## Abstract (arXiv)

The tremendous recent progress in analyzing the training dynamics of overparameterized neural networks has primarily focused on wide networks and therefore does not sufficiently address the role of depth in deep learning. In this work, we present the first trainability guarantee of infinitely deep but narrow neural networks. We study the infinite-depth limit of a multilayer perceptron (MLP) with a specific initialization and establish a trainability guarantee using the NTK theory. We then extend the analysis to an infinitely deep convolutional neural network (CNN) and perform brief experiments.