# Deep kernel processes

arXiv: https://arxiv.org/abs/2010.01590 · HTML: https://mlsanity.com/paper/2010.01590/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Oct 2020.

## Built on by (most cited first, 30 of 34)

- [The Low-Rank Simplicity Bias in Deep Networks](https://mlsanity.com/paper/2103.10427/) (2021)
- [Bayesian Neural Network Priors Revisited](https://mlsanity.com/paper/2102.06571/) (2021)
- [Bayesian Low-rank Adaptation for Large Language Models](https://mlsanity.com/paper/2308.13111/) (2023)
- [Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AI](https://mlsanity.com/paper/2402.00809/) (2024)
- [Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNs](https://mlsanity.com/paper/2112.15383/) (2021)
- [Uncertainty quantification in fine-tuned LLMs using LoRA ensembles](https://mlsanity.com/paper/2402.12264/) (2024)
- [Bayesian Reward Models for LLM Alignment](https://mlsanity.com/paper/2402.13210/) (2024)
- [Precise characterization of the prior predictive distribution of deep ReLU networks](https://mlsanity.com/paper/2106.06615/) (2021)
- [Random initialisations performing above chance and how to find them](https://mlsanity.com/paper/2209.07509/) (2022)
- [The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspective](https://mlsanity.com/paper/2106.06529/) (2021)
- [The Empirical Impact of Neural Parameter Symmetries, or Lack Thereof](https://mlsanity.com/paper/2405.20231/) (2024)
- [A theory of representation learning gives a deep generalisation of kernel methods](https://mlsanity.com/paper/2108.13097/) (2021)
- [Evolution of Neural Tangent Kernels under Benign and Adversarial Training](https://mlsanity.com/paper/2210.12030/) (2022)
- [Towards Understanding Inductive Bias in Transformers: A View From Infinity](https://mlsanity.com/paper/2402.05173/) (2024)
- [Wide Bayesian neural networks have a simple weight posterior: theory and accelerated sampling](https://mlsanity.com/paper/2206.07673/) (2022)
- [On the detrimental effect of invariances in the likelihood for variational inference](https://mlsanity.com/paper/2209.07157/) (2022)
- [Depth induces scale-averaging in overparameterized linear Bayesian neural networks](https://mlsanity.com/paper/2111.11954/) (2021)
- [A variational approximate posterior for the deep Wishart process](https://mlsanity.com/paper/2107.10125/) (2021)
- [Guided Deep Kernel Learning](https://mlsanity.com/paper/2302.09574/) (2023)
- [Collapsed Inference for Bayesian Deep Learning](https://mlsanity.com/paper/2306.09686/) (2023)
- [Neural Hilbert Ladders: Multi-Layer Neural Networks in Function Space](https://mlsanity.com/paper/2307.01177/) (2023)
- [Self-Attention through Kernel-Eigen Pair Sparse Variational Gaussian Processes](https://mlsanity.com/paper/2402.01476/) (2024)
- [Minimal Ranks, Maximum Confidence: Parameter-efficient Uncertainty Quantification for LoRA](https://mlsanity.com/paper/2502.12122/) (2025)
- [Posterior Inference on Shallow Infinitely Wide Bayesian Neural Networks under Weights with Unbounded Variance](https://mlsanity.com/paper/2305.10664/) (2023)
- [An Improved Variational Approximate Posterior for the Deep Wishart Process](https://mlsanity.com/paper/2305.14454/) (2023)
- [Convolutional Deep Kernel Machines](https://mlsanity.com/paper/2309.09814/) (2023)
- [Inducing Gaussian Process Networks](https://mlsanity.com/paper/2204.09889/) (2022)
- [Variational Elliptical Processes](https://mlsanity.com/paper/2311.12566/) (2023)
- [Flexible Infinite-Width Graph Convolutional Neural Networks](https://mlsanity.com/paper/2402.06525/) (2024)
- [Deep Kernel Posterior Learning under Infinite Variance Prior Weights](https://mlsanity.com/paper/2410.01284/) (2024)

## Abstract (arXiv)

We define deep kernel processes in which positive definite Gram matrices are progressively transformed by nonlinear kernel functions and by sampling from (inverse) Wishart distributions. Remarkably, we find that deep Gaussian processes (DGPs), Bayesian neural networks (BNNs), infinite BNNs, and infinite BNNs with bottlenecks can all be written as deep kernel processes. For DGPs the equivalence arises because the Gram matrix formed by the inner product of features is Wishart distributed, and as we show, standard isotropic kernels can be written entirely in terms of this Gram matrix -- we do not need knowledge of the underlying features. We define a tractable deep kernel process, the deep inverse Wishart process, and give a doubly-stochastic inducing-point variational inference scheme that operates on the Gram matrices, not on the features, as in DGPs. We show that the deep inverse Wishart process gives superior performance to DGPs and infinite BNNs on standard fully-connected baselines.