mlsanityData to 14 Oct 2026

Spectral Analysis of the Neural Tangent Kernel for Deep Residual Networks

arXiv 2104.03093 ↗First posted 7 Apr 2021cs.LGcited by 21

Not on the map. Why?

Abstract (arXiv)

Deep residual network architectures have been shown to achieve superior accuracy over classical feed-forward networks, yet their success is still not fully understood. Focusing on massively over-parameterized, fully connected residual networks with ReLU activation through their respective neural tangent kernels (ResNTK), we provide here a spectral analysis of these kernels. Specifically, we show that, much like NTK for fully connected networks (FC-NTK), for input distributed uniformly on the hypersphere mathbbSd-1, the eigenfunctions of ResNTK are the spherical harmonics and the eigenvalues decay polynomially with frequency k as k-d. These in turn imply that the set of functions in their Reproducing Kernel Hilbert Space are identical to those of FC-NTK, and consequently also to those of the Laplace kernel. We further show, by drawing on the analogy to the Laplace kernel, that depending on the choice of a hyper-parameter that balances between the skip and residual connections ResNTK can either become spiky with depth, as with FC-NTK, or maintain a stable shape.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 42 of its 56 references that are in the map. First posted on arXiv 7 Apr 2021.

Builds on

42 of 56 references.

References · 42

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  3. 2019Searching for MobileNetV3cited by 1,994
  4. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  5. 2018On the Spectral Bias of Neural Networkscited by 1,341
  6. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  7. 2020Designing Network Design Spacescited by 960
  8. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 943
  9. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  10. 2018On Lazy Training in Differentiable Programmingcited by 675
  11. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
  12. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
30 more
  1. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  2. 2019Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networkscited by 368
  3. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 198
  4. 2018Gaussian Processes and Kernel Methods: A Review on Connections and Equivalencescited by 192
  5. 2018Just Interpolate: Kernel "Ridgeless" Regression Can Generalizecited by 190
  6. 2019On the Inductive Bias of Neural Tangent Kernelscited by 177
  7. 2019Towards Understanding the Spectral Bias of Deep Learningcited by 174
  8. 2017The Shattered Gradients Problem: If resnets are the answer, then what is the question?cited by 157
  9. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 157
  10. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
  11. 2020Frequency Bias in Neural Networks for Input of Non-Uniform Densitycited by 140
  12. 2019The Convergence Rate of Neural Networks for Learned Functions of Different Frequenciescited by 135
  13. 2019On the Power and Limitations of Random Features for Understanding Neural Networkscited by 103
  14. 2019Harnessing the Power of Infinitely Wide Deep Nets on Small-data Taskscited by 87
  15. 2020Deep Equals Shallow for ReLU Networks in Kernel Regimescited by 85
  16. 2019Learning to Optimize Multigrid PDE Solverscited by 79
  17. 2020Learning Parities with Neural Networkscited by 78
  18. 2020Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networkscited by 77
  19. 2020On the Similarity between the Laplace and Neural Tangent Kernelscited by 76
  20. 2020Deep Neural Tangent Kernel and Laplace Kernel Have the Same RKHScited by 74
  21. 2020Why Do Deep Residual Networks Generalize Better than Deep Feedforward Networks? -- A Neural Tangent Kernel Perspectivecited by 60
  22. 2020Stable ResNetcited by 49
  23. 2021Quantifying the Benefit of Using Differentiable Learning over Tangent Kernelscited by 41
  24. 2020Optimal Rates for Averaged Stochastic Gradient Descent under Neural Tangent Kernel Regimecited by 35
  25. 2019On Learning Over-parameterized Neural Networks: A Functional Approximation Perspectivecited by 28
  26. 2020Kernel-Based Smoothness Analysis of Residual Networkscited by 19
  27. 2019Towards Understanding the Importance of Shortcut Connections in Residual Networkscited by 18
  28. 2020The Neural Tangent Link Between CNN Denoisers and Non-Local Filterscited by 9
  29. 2019On the Multiple Descent of Minimum-Norm Interpolants and Restricted Lower Isometry of Kernelscited by 7
  30. 2020Interpolation and Learning with Scale Dependent Kernelscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 3; 2023: 5; 2024: 3; 2025: 4; 2026: 5.

21 papers.

Sort:

By area:

  1. 2023Generalization in Kernel Regression Under Realistic Assumptionscited by 27
  2. 2025The Universal Weight Subspace Hypothesiscited by 24
  3. 2021Learning Curves for Continual Learning in Neural Networks: Self-Knowledge Transfer and Forgettingcited by 14
  4. 2022Generalization Properties of NAS under Activation and Skip Connection Searchcited by 14
  5. 2022A Kernel Perspective of Skip Connections in Convolutional Networkscited by 12
  6. 2023Controlling the Inductive Bias of Wide Neural Networks by Modifying the Kernel's Spectrumcited by 9
  7. 2022Extrapolation and Spectral Bias of Neural Nets with Hadamard Product: a Polynomial Net Studycited by 7
  8. 2023Better NTK Conditioning: A Free Lunch from (ReLU) Nonlinear Activation in Wide Neural Networkscited by 6
  9. 2023Generalization Ability of Wide Residual Networkscited by 5
  10. 2026Richer Bayesian Last Layers with Subsampled NTK Featurescited by 4
  11. 2024Generalization of Scaled Deep ResNets in the Mean-Field Regimecited by 3
  12. 2024HOIN: High-Order Implicit Neural Representationscited by 3
9 more
  1. 2025Divergence of Empirical Neural Tangent Kernel in Classification Problemscited by 3
  2. 2025Beyond ReLU: How Activations Affect Neural Kernels and Random Wide Networkscited by 3
  3. 2023Sparse Gaussian Processes with Spherical Harmonic Features Revisitedcited by 2
  4. 2024FedDr+: Stabilizing Dot-regression with Global Feature Distillation for Federated Learningcited by 2
  5. 2025Condition Numbers and Eigenvalue Spectra of Shallow Networks on Spherescited by 2
  6. 2026Gradient Residual Connectionscited by 0
  7. 2026Global Federated Learning Strategies for Building Efficient Personalized Modelscited by 0
  8. 2026The role of parameter Jacobians in the stability of network outputscited by 0
  9. 2026Latent-MoE: Domain-Aware Mixture-of-Experts for PDEs with Multi-Regime Physicscited by 0