mlsanityData to 14 Oct 2026

A Kernel Perspective of Skip Connections in Convolutional Networks

arXiv 2211.14810 ↗First posted 27 Nov 2022cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

Over-parameterized residual networks (ResNets) are amongst the most successful convolutional neural architectures for image processing. Here we study their properties through their Gaussian Process and Neural Tangent kernels. We derive explicit formulas for these kernels, analyze their spectra, and provide bounds on their implied condition numbers. Our results indicate that (1) with ReLU activation, the eigenvalues of these residual kernels decay polynomially at a similar rate compared to the same kernels when skip connections are not used, thus maintaining a similar frequency bias; (2) however, residual kernels are more locally biased. Our analysis further shows that the matrices obtained by these residual kernels yield favorable condition numbers at finite depths than those obtained without the skip connections, enabling therefore faster convergence of training with gradient descent.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 32 of its 48 references that are in the map. First posted on arXiv 27 Nov 2022.

Builds on

32 of 48 references.

References · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  3. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  4. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  5. 2018On Lazy Training in Differentiable Programmingcited by 675
  6. 2017Deep Neural Networks as Gaussian Processescited by 622
  7. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
  8. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  9. 2017Understanding the Effective Receptive Field in Deep Convolutional Neural Networkscited by 553
  10. 2016Benefits of depth in neural networkscited by 250
  11. 2018Gaussian Processes and Kernel Methods: A Review on Connections and Equivalencescited by 192
  12. 2019On the Inductive Bias of Neural Tangent Kernelscited by 177
20 more
  1. 2021Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspectivecited by 169
  2. 2017The Shattered Gradients Problem: If resnets are the answer, then what is the question?cited by 157
  3. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 157
  4. 2016Why and When Can Deep -- but Not Shallow -- Networks Avoid the Curse of Dimensionality: a Reviewcited by 155
  5. 2020Tensor Programs II: Neural Tangent Kernel for Any Architecturecited by 143
  6. 2020Frequency Bias in Neural Networks for Input of Non-Uniform Densitycited by 140
  7. 2016Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivitycited by 138
  8. 2019Harnessing the Power of Infinitely Wide Deep Nets on Small-data Taskscited by 87
  9. 2020Deep Equals Shallow for ReLU Networks in Kernel Regimescited by 85
  10. 2020On the Similarity between the Laplace and Neural Tangent Kernelscited by 76
  11. 2020Deep Neural Tangent Kernel and Laplace Kernel Have the Same RKHScited by 74
  12. 2020Why Do Deep Residual Networks Generalize Better than Deep Feedforward Networks? -- A Neural Tangent Kernel Perspectivecited by 60
  13. 2021Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamicscited by 57
  14. 2021Locality defeats the curse of dimensionality in convolutional teacher-student scenarioscited by 26
  15. 2021Spectral Analysis of the Neural Tangent Kernel for Deep Residual Networkscited by 21
  16. 2020Computational Separation Between Convolutional and Fully-Connected Networkscited by 20
  17. 2020Kernel-Based Smoothness Analysis of Residual Networkscited by 19
  18. 2021Eigenspace Restructuring: a Principle of Space and Frequency in Neural Networkscited by 17
  19. 2022On the Spectral Bias of Convolutional Neural Tangent and Gaussian Process Kernelscited by 16
  20. 2017The exploding gradient problem demystified - definition, prevalence, impact, origin, tradeoffs, and solutionscited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 2; 2025: 2; 2026: 1.

12 papers.

  1. 2023Generalization in Kernel Regression Under Realistic Assumptionscited by 27
  2. 2023Controlling the Inductive Bias of Wide Neural Networks by Modifying the Kernel's Spectrumcited by 9
  3. 2023Theoretical Analysis of Robust Overfitting for Wide DNNs: An NTK Approachcited by 9
  4. 2023Expressibility-induced Concentration of Quantum Neural Tangent Kernelscited by 7
  5. 2023Stimulative Training++: Go Beyond The Performance Limits of Residual Networkscited by 6
  6. 2023Field theory for optimal signal propagation in ResNetscited by 6
  7. 2024Can Kernel Methods Explain How the Data Affects Neural Collapse?cited by 4
  8. 2023Boosting Residual Networks with Group Knowledgecited by 3
  9. 2024Generalization of Scaled Deep ResNets in the Mean-Field Regimecited by 3
  10. 2025Beyond ReLU: How Activations Affect Neural Kernels and Random Wide Networkscited by 3
  11. 2025Querying Kernel Methods Suffices for Reconstructing their Training Datacited by 0
  12. 2026Latent-MoE: Domain-Aware Mixture-of-Experts for PDEs with Multi-Regime Physicscited by 0