mlsanityData to 14 Oct 2026

Neural Tangent Kernel Analysis of Deep Narrow Neural Networks

arXiv 2202.02981 ↗First posted 7 Feb 2022cs.LG math.OC stat.MLcited by 11

Not on the map. Why?

Abstract (arXiv)

The tremendous recent progress in analyzing the training dynamics of overparameterized neural networks has primarily focused on wide networks and therefore does not sufficiently address the role of depth in deep learning. In this work, we present the first trainability guarantee of infinitely deep but narrow neural networks. We study the infinite-depth limit of a multilayer perceptron (MLP) with a specific initialization and establish a trainability guarantee using the NTK theory. We then extend the analysis to an infinitely deep convolutional neural network (CNN) and perform brief experiments.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 11 of its 74 references that are in the map. First posted on arXiv 7 Feb 2022.

Builds on

11 of 74 references.

References · 11

  1. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  3. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  4. 2015Highway Networkscited by 260
  5. 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 149
  6. 2017Approximating Continuous Functions by ReLU Nets of Minimal Widthcited by 122
  7. 2020A Rigorous Framework for the Mean Field Limit of Multilayer Neural Networkscited by 68
  8. 2020On the infinite width limit of neural networks with a standard parameterizationcited by 38
  9. 2021Overparameterization of deep ResNet: zero loss and mean-field analysiscited by 24
  10. 2021On the Global Convergence of Gradient Descent for multi-layer ResNets in the mean-field regimecited by 12
  11. 2019Disentangling feature and lazy training in deep neural networkscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 3; 2024: 2; 2025: 4; 2026: 1.

11 papers.

  1. 2023How many Neurons do we need? A refined Analysis for Shallow Networks trained with Gradient Descentcited by 10
  2. 2024Information-Theoretic Generalization Bounds for Deep Neural Networkscited by 10
  3. 2023Theoretical Analysis of Robust Overfitting for Wide DNNs: An NTK Approachcited by 9
  4. 2022Approximation results for Gradient Descent trained Shallow Neural Networks in 1dcited by 7
  5. 2023Approximation Results for Gradient Descent trained Neural Networkscited by 2
  6. 2025Non-vacuous Generalization Bounds for Deep Neural Networks without any modification to the trained modelscited by 2
  7. 2024Approximation and Gradient Descent Training with Neural Networkscited by 1
  8. 2025Depth-induced NTK: Bridging Over-parameterized Neural Networks and Deep Neural Kernelscited by 1
  9. 2025Compact Vision Transformer by Reduction of Kernel Complexitycited by 0
  10. 2025Revisiting the Neural Tangent Kernel: the role of large width and depthcited by 0
  11. 2026A Gaussian Comparison Theorem for Training Dynamics in Machine Learningcited by 0