# Steps Toward Deep Kernel Methods from Infinite Neural Networks

arXiv: https://arxiv.org/abs/1508.05133 · HTML: https://mlsanity.com/paper/1508.05133/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Aug 2015.

## Built on by (most cited first, 28 of 28)

- [Priors in Bayesian Deep Learning: A Review](https://mlsanity.com/paper/2105.06868/) (2021)
- [Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamics](https://mlsanity.com/paper/2105.03703/) (2021)
- [Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shaping](https://mlsanity.com/paper/2110.01765/) (2021)
- [A Fast, Well-Founded Approximation to the Empirical Neural Tangent Kernel](https://mlsanity.com/paper/2206.12543/) (2022)
- [Batch Normalization Orthogonalizes Representations in Deep Random Networks](https://mlsanity.com/paper/2106.03970/) (2021)
- [Non-asymptotic approximations of neural networks by Gaussian processes](https://mlsanity.com/paper/2102.08668/) (2021)
- [Deep Stable neural networks: large-width asymptotics and convergence rates](https://mlsanity.com/paper/2108.02316/) (2021)
- [Imitating Deep Learning Dynamics via Locally Elastic Stochastic Differential Equations](https://mlsanity.com/paper/2110.05960/) (2021)
- [SHAP values via sparse Fourier representation](https://mlsanity.com/paper/2410.06300/) (2024)
- [Centered Self-Attention Layers](https://mlsanity.com/paper/2306.01610/) (2023)
- [NTK-Guided Few-Shot Class Incremental Learning](https://mlsanity.com/paper/2403.12486/) (2024)
- [Connecting NTK and NNGP: A Unified Theoretical Framework for Wide Neural Network Learning Dynamics](https://mlsanity.com/paper/2309.04522/) (2023)
- [Learning Representation from Neural Fisher Kernel with Low-rank Approximation](https://mlsanity.com/paper/2202.01944/) (2022)
- [Neural Hilbert Ladders: Multi-Layer Neural Networks in Function Space](https://mlsanity.com/paper/2307.01177/) (2023)
- [Descriptive Kernel Convolution Network with Improved Random Walk Kernel](https://mlsanity.com/paper/2402.06087/) (2024)
- [Deep Horseshoe Gaussian Processes](https://mlsanity.com/paper/2403.01737/) (2024)
- [Optimal Activation Functions for the Random Features Regression Model](https://mlsanity.com/paper/2206.01332/) (2022)
- [Correlated Weights in Infinite Limits of Deep Convolutional Neural Networks](https://mlsanity.com/paper/2101.04097/) (2021)
- [Bayesian Neural Network Language Modeling for Speech Recognition](https://mlsanity.com/paper/2208.13259/) (2022)
- [Neural networks in non-metric spaces](https://mlsanity.com/paper/2406.09310/) (2024)
- [Finite Neural Networks as Mixtures of Gaussian Processes: From Provable Error Bounds to Prior Selection](https://mlsanity.com/paper/2407.18707/) (2024)
- [MoSE: Unveiling Structural Patterns in Graphs via Mixture of Subgraph Experts](https://mlsanity.com/paper/2509.09337/) (2025)
- [Infinitely Wide Tensor Networks as Gaussian Process](https://mlsanity.com/paper/2101.02333/) (2021)
- [Implicit Acceleration and Feature Learning in Infinitely Wide Neural Networks with Bottlenecks](https://mlsanity.com/paper/2107.00364/) (2021)
- [A connection between probability, physics and neural networks](https://mlsanity.com/paper/2209.12737/) (2022)
- [Weighted Neural Tangent Kernel: A Generalized and Improved Network-Induced Kernel](https://mlsanity.com/paper/2103.11558/) (2021)
- [Neural Generalization of Multiple Kernel Learning](https://mlsanity.com/paper/2102.13337/) (2021)
- [A Simple Algorithm For Scaling Up Kernel Methods](https://mlsanity.com/paper/2301.11414/) (2023)

## Abstract (arXiv)

Contemporary deep neural networks exhibit impressive results on practical problems. These networks generalize well although their inherent capacity may extend significantly beyond the number of training examples. We analyze this behavior in the context of deep, infinite neural networks. We show that deep infinite layers are naturally aligned with Gaussian processes and kernel methods, and devise stochastic kernels that encode the information of these networks. We show that stability results apply despite the size, offering an explanation for their empirical success.