# A variational approximate posterior for the deep Wishart process

arXiv: https://arxiv.org/abs/2107.10125 · HTML: https://mlsanity.com/paper/2107.10125/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 39 references that are in the map.
- First posted on arXiv 21 Jul 2021.

## Builds on (most cited first, 17 of 17)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Deep Neural Networks as Gaussian Processes](https://mlsanity.com/paper/1711.00165/) (2017)
- [Gaussian Process Behaviour in Wide Deep Neural Networks](https://mlsanity.com/paper/1804.11271/) (2018)
- [Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes](https://mlsanity.com/paper/1810.05148/) (2018)
- [Doubly Stochastic Variational Inference for Deep Gaussian Processes](https://mlsanity.com/paper/1705.08933/) (2017)
- [Deep Convolutional Networks as shallow Gaussian Processes](https://mlsanity.com/paper/1808.05587/) (2018)
- [Functional Variational Bayesian Neural Networks](https://mlsanity.com/paper/1903.05779/) (2019)
- [Sticking the Landing: Simple, Lower-Variance Gradient Estimators for Variational Inference](https://mlsanity.com/paper/1703.09194/) (2017)
- [Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processes](https://mlsanity.com/paper/2005.08140/) (2020)
- [Variational Implicit Processes](https://mlsanity.com/paper/1806.02390/) (2018)
- [Why bigger is not always better: on finite and infinite neural networks](https://mlsanity.com/paper/1910.08013/) (2019)
- [Understanding Variational Inference in Function-Space](https://mlsanity.com/paper/2011.09421/) (2020)
- [Deep kernel processes](https://mlsanity.com/paper/2010.01590/) (2020)
- [Avoiding pathologies in very deep networks](https://mlsanity.com/paper/1402.5836/) (2014)
- [Scalable Bayesian dynamic covariance modeling with variational Wishart and inverse Wishart processes](https://mlsanity.com/paper/1906.09360/) (2019)
- [Stochastic Differential Equations with Variational Wishart Diffusions](https://mlsanity.com/paper/2006.14895/) (2020)

## Built on by (most cited first, 6 of 6)

- [Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AI](https://mlsanity.com/paper/2402.00809/) (2024)
- [A theory of representation learning gives a deep generalisation of kernel methods](https://mlsanity.com/paper/2108.13097/) (2021)
- [Guided Deep Kernel Learning](https://mlsanity.com/paper/2302.09574/) (2023)
- [An Improved Variational Approximate Posterior for the Deep Wishart Process](https://mlsanity.com/paper/2305.14454/) (2023)
- [Convolutional Deep Kernel Machines](https://mlsanity.com/paper/2309.09814/) (2023)
- [Stochastic Kernel Regularisation Improves Generalisation in Deep Kernel Machines](https://mlsanity.com/paper/2410.06171/) (2024)

## Abstract (arXiv)

Recent work introduced deep kernel processes as an entirely kernel-based alternative to NNs (Aitchison et al. 2020). Deep kernel processes flexibly learn good top-layer representations by alternately sampling the kernel from a distribution over positive semi-definite matrices and performing nonlinear transformations. A particular deep kernel process, the deep Wishart process (DWP), is of particular interest because its prior can be made equivalent to deep Gaussian process (DGP) priors for kernels that can be expressed entirely in terms of Gram matrices. However, inference in DWPs has not yet been possible due to the lack of sufficiently flexible distributions over positive semi-definite matrices. Here, we give a novel approach to obtaining flexible distributions over positive semi-definite matrices by generalising the Bartlett decomposition of the Wishart probability density. We use this new distribution to develop an approximate posterior for the DWP that includes dependency across layers. We develop a doubly-stochastic inducing-point inference scheme for the DWP and show experimentally that inference in the DWP can improve performance over doing inference in a DGP with the equivalent prior.