# Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processes

arXiv: https://arxiv.org/abs/2005.08140 · HTML: https://mlsanity.com/paper/2005.08140/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 May 2020.

## Built on by (most cited first, 30 of 43)

- [Bayesian Neural Network Priors Revisited](https://mlsanity.com/paper/2102.06571/) (2021)
- [Bayesian Low-rank Adaptation for Large Language Models](https://mlsanity.com/paper/2308.13111/) (2023)
- [Priors in Bayesian Deep Learning: A Review](https://mlsanity.com/paper/2105.06868/) (2021)
- [The Promises and Pitfalls of Deep Kernel Learning](https://mlsanity.com/paper/2102.12108/) (2021)
- [Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AI](https://mlsanity.com/paper/2402.00809/) (2024)
- [Invariance Learning in Deep Neural Networks with Differentiable Laplace Approximations](https://mlsanity.com/paper/2202.10638/) (2022)
- [Tractable Function-Space Variational Inference in Bayesian Neural Networks](https://mlsanity.com/paper/2312.17199/) (2023)
- [Uncertainty quantification in fine-tuned LLMs using LoRA ensembles](https://mlsanity.com/paper/2402.12264/) (2024)
- [Data augmentation in Bayesian neural networks and the cold posterior effect](https://mlsanity.com/paper/2106.05586/) (2021)
- [Bayesian Reward Models for LLM Alignment](https://mlsanity.com/paper/2402.13210/) (2024)
- [Gradient Descent on Neurons and its Link to Approximate Second-Order Optimization](https://mlsanity.com/paper/2201.12250/) (2022)
- [Adapting the Linearised Laplace Model Evidence for Modern Deep Learning](https://mlsanity.com/paper/2206.08900/) (2022)
- [Deep Neural Networks as Point Estimates for Deep Gaussian Processes](https://mlsanity.com/paper/2105.04504/) (2021)
- [Learning Layer-wise Equivariances Automatically using Gradients](https://mlsanity.com/paper/2310.06131/) (2023)
- [The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspective](https://mlsanity.com/paper/2106.06529/) (2021)
- [Last Layer Marginal Likelihood for Invariance Learning](https://mlsanity.com/paper/2106.07512/) (2021)
- [Function-Space Regularization in Neural Networks: A Probabilistic Perspective](https://mlsanity.com/paper/2312.17162/) (2023)
- [Sparse Uncertainty Representation in Deep Learning with Inducing Weights](https://mlsanity.com/paper/2105.14594/) (2021)
- [Learning Invariant Weights in Neural Networks](https://mlsanity.com/paper/2202.12439/) (2022)
- [Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Model](https://mlsanity.com/paper/2503.04543/) (2025)
- [Deep neural networks with dependent weights: Gaussian Process mixture limit, heavy tails, sparsity and compressibility](https://mlsanity.com/paper/2205.08187/) (2022)
- [The Peril of Popular Deep Learning Uncertainty Estimation Methods](https://mlsanity.com/paper/2112.05000/) (2021)
- [Stochastic Marginal Likelihood Gradients using Neural Tangent Kernels](https://mlsanity.com/paper/2306.03968/) (2023)
- [Wide Bayesian neural networks have a simple weight posterior: theory and accelerated sampling](https://mlsanity.com/paper/2206.07673/) (2022)
- [Incorporating Unlabelled Data into Bayesian Neural Networks](https://mlsanity.com/paper/2304.01762/) (2023)
- [Are you using test log-likelihood correctly?](https://mlsanity.com/paper/2212.00219/) (2022)
- [A variational approximate posterior for the deep Wishart process](https://mlsanity.com/paper/2107.10125/) (2021)
- [Neural Operator Variational Inference based on Regularized Stein Discrepancy for Deep Gaussian Processes](https://mlsanity.com/paper/2309.12658/) (2023)
- [Robustness to corruption in pre-trained Bayesian neural networks](https://mlsanity.com/paper/2206.12361/) (2022)
- [Variational Inference of overparameterized Bayesian Neural Networks: a theoretical and empirical study](https://mlsanity.com/paper/2207.03859/) (2022)

## Abstract (arXiv)

We consider the optimal approximate posterior over the top-layer weights in a Bayesian neural network for regression, and show that it exhibits strong dependencies on the lower-layer weights. We adapt this result to develop a correlated approximate posterior over the weights at all layers in a Bayesian neural network. We extend this approach to deep Gaussian processes, unifying inference in the two model classes. Our approximate posterior uses learned "global" inducing points, which are defined only at the input layer and propagated through the network to obtain inducing inputs at subsequent layers. By contrast, standard, "local", inducing point methods from the deep Gaussian process literature optimise a separate set of inducing inputs at every layer, and thus do not model correlations across layers. Our method gives state-of-the-art performance for a variational Bayesian method, without data augmentation or tempering, on CIFAR-10 of 86.7%, which is comparable to SGMCMC without tempering but with data augmentation (88% in Wenzel et al. 2020).