mlsanityData to 7 Oct 2026

Deep kernel processes

arXiv 2010.01590 ↗First posted 4 Oct 2020stat.ML cs.LGcited by 34

Not on the map. Why?

Abstract (arXiv)

We define deep kernel processes in which positive definite Gram matrices are progressively transformed by nonlinear kernel functions and by sampling from (inverse) Wishart distributions. Remarkably, we find that deep Gaussian processes (DGPs), Bayesian neural networks (BNNs), infinite BNNs, and infinite BNNs with bottlenecks can all be written as deep kernel processes. For DGPs the equivalence arises because the Gram matrix formed by the inner product of features is Wishart distributed, and as we show, standard isotropic kernels can be written entirely in terms of this Gram matrix -- we do not need knowledge of the underlying features. We define a tractable deep kernel process, the deep inverse Wishart process, and give a doubly-stochastic inducing-point variational inference scheme that operates on the Gram matrices, not on the features, as in DGPs. We show that the deep inverse Wishart process gives superior performance to DGPs and infinite BNNs on standard fully-connected baselines.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). First posted on arXiv 4 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 5; 2023: 8; 2024: 10; 2025: 1; 2026: 2.

34 papers.

Sort:

By area:

  1. 2021The Low-Rank Simplicity Bias in Deep Networkscited by 150
  2. 2021Bayesian Neural Network Priors Revisitedcited by 121
  3. 2023Bayesian Low-rank Adaptation for Large Language Modelscited by 112
  4. 2024Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AIcited by 60
  5. 2021Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNscited by 53
  6. 2024Uncertainty quantification in fine-tuned LLMs using LoRA ensemblescited by 35
  7. 2024Bayesian Reward Models for LLM Alignmentcited by 35
  8. 2021Precise characterization of the prior predictive distribution of deep ReLU networkscited by 28
  9. 2022Random initialisations performing above chance and how to find themcited by 23
  10. 2021The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspectivecited by 21
  11. 2024The Empirical Impact of Neural Parameter Symmetries, or Lack Thereofcited by 21
  12. 2021A theory of representation learning gives a deep generalisation of kernel methodscited by 15
22 more
  1. 2022Evolution of Neural Tangent Kernels under Benign and Adversarial Trainingcited by 14
  2. 2024Towards Understanding Inductive Bias in Transformers: A View From Infinitycited by 13
  3. 2022Wide Bayesian neural networks have a simple weight posterior: theory and accelerated samplingcited by 9
  4. 2022On the detrimental effect of invariances in the likelihood for variational inferencecited by 8
  5. 2021Depth induces scale-averaging in overparameterized linear Bayesian neural networkscited by 7
  6. 2021A variational approximate posterior for the deep Wishart processcited by 6
  7. 2023Guided Deep Kernel Learningcited by 6
  8. 2023Collapsed Inference for Bayesian Deep Learningcited by 6
  9. 2023Neural Hilbert Ladders: Multi-Layer Neural Networks in Function Spacecited by 5
  10. 2025Minimal Ranks, Maximum Confidence: Parameter-efficient Uncertainty Quantification for LoRAcited by 5
  11. 2024Self-Attention through Kernel-Eigen Pair Sparse Variational Gaussian Processescited by 4
  12. 2023Posterior Inference on Shallow Infinitely Wide Bayesian Neural Networks under Weights with Unbounded Variancecited by 3
  13. 2023An Improved Variational Approximate Posterior for the Deep Wishart Processcited by 3
  14. 2023Convolutional Deep Kernel Machinescited by 3
  15. 2022Inducing Gaussian Process Networkscited by 1
  16. 2023Variational Elliptical Processescited by 1
  17. 2024Flexible Infinite-Width Graph Convolutional Neural Networkscited by 0
  18. 2024Deep Kernel Posterior Learning under Infinite Variance Prior Weightscited by 0
  19. 2024Stochastic Kernel Regularisation Improves Generalisation in Deep Kernel Machinescited by 0
  20. 2024Deep Q-Exponential Processescited by 0
  21. 2026No More DeLuLu: Physics-Inspired Kernel Networks for Geometrically-Grounded Neural Computationcited by 0
  22. 2026Bayesian Fine-tuning in Projected Subspacescited by 0