mlsanityData to 14 Oct 2026

Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processes

arXiv 2005.08140 ↗First posted 17 May 2020stat.ML cs.LGcited by 43

Not on the map. Why?

Abstract (arXiv)

We consider the optimal approximate posterior over the top-layer weights in a Bayesian neural network for regression, and show that it exhibits strong dependencies on the lower-layer weights. We adapt this result to develop a correlated approximate posterior over the weights at all layers in a Bayesian neural network. We extend this approach to deep Gaussian processes, unifying inference in the two model classes. Our approximate posterior uses learned "global" inducing points, which are defined only at the input layer and propagated through the network to obtain inducing inputs at subsequent layers. By contrast, standard, "local", inducing point methods from the deep Gaussian process literature optimise a separate set of inducing inputs at every layer, and thus do not model correlations across layers. Our method gives state-of-the-art performance for a variational Bayesian method, without data augmentation or tempering, on CIFAR-10 of 86.7%, which is comparable to SGMCMC without tempering but with data augmentation (88% in Wenzel et al. 2020).

Key facts. Cited by 43 papers in the map (data through 14 Oct 2026). First posted on arXiv 17 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 13; 2022: 11; 2023: 12; 2024: 3; 2025: 3; 2026: 1.

43 papers.

Sort:

By area:

  1. 2021Bayesian Neural Network Priors Revisitedcited by 121
  2. 2023Bayesian Low-rank Adaptation for Large Language Modelscited by 115
  3. 2021Priors in Bayesian Deep Learning: A Reviewcited by 101
  4. 2021The Promises and Pitfalls of Deep Kernel Learningcited by 81
  5. 2024Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AIcited by 62
  6. 2022Invariance Learning in Deep Neural Networks with Differentiable Laplace Approximationscited by 42
  7. 2023Tractable Function-Space Variational Inference in Bayesian Neural Networkscited by 41
  8. 2024Uncertainty quantification in fine-tuned LLMs using LoRA ensemblescited by 37
  9. 2021Data augmentation in Bayesian neural networks and the cold posterior effectcited by 36
  10. 2024Bayesian Reward Models for LLM Alignmentcited by 35
  11. 2022Gradient Descent on Neurons and its Link to Approximate Second-Order Optimizationcited by 30
  12. 2022Adapting the Linearised Laplace Model Evidence for Modern Deep Learningcited by 29
31 more
  1. 2021Deep Neural Networks as Point Estimates for Deep Gaussian Processescited by 26
  2. 2023Learning Layer-wise Equivariances Automatically using Gradientscited by 23
  3. 2021The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspectivecited by 21
  4. 2021Last Layer Marginal Likelihood for Invariance Learningcited by 21
  5. 2023Function-Space Regularization in Neural Networks: A Probabilistic Perspectivecited by 20
  6. 2021Sparse Uncertainty Representation in Deep Learning with Inducing Weightscited by 16
  7. 2022Learning Invariant Weights in Neural Networkscited by 16
  8. 2025Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Modelcited by 16
  9. 2022Deep neural networks with dependent weights: Gaussian Process mixture limit, heavy tails, sparsity and compressibilitycited by 13
  10. 2021The Peril of Popular Deep Learning Uncertainty Estimation Methodscited by 12
  11. 2023Stochastic Marginal Likelihood Gradients using Neural Tangent Kernelscited by 10
  12. 2022Wide Bayesian neural networks have a simple weight posterior: theory and accelerated samplingcited by 9
  13. 2023Incorporating Unlabelled Data into Bayesian Neural Networkscited by 9
  14. 2022Are you using test log-likelihood correctly?cited by 8
  15. 2021A variational approximate posterior for the deep Wishart processcited by 6
  16. 2023Neural Operator Variational Inference based on Regularized Stein Discrepancy for Deep Gaussian Processescited by 6
  17. 2022Robustness to corruption in pre-trained Bayesian neural networkscited by 5
  18. 2022Variational Inference of overparameterized Bayesian Neural Networks: a theoretical and empirical studycited by 5
  19. 2023An Improved Variational Approximate Posterior for the Deep Wishart Processcited by 3
  20. 2023Amortised Inference in Neural Networks for Small-Scale Probabilistic Meta-Learningcited by 3
  21. 2022Generalised Co-Salient Object Detectioncited by 2
  22. 2023Evaluation of machine learning architectures on the quantification of epistemic and aleatoric uncertainties in complex dynamical systemscited by 2
  23. 2025Sparse Gaussian Neural Processescited by 2
  24. 2021Variational Laplace for Bayesian neural networkscited by 1
  25. 2021Conditional Deep Gaussian Processes: empirical Bayes hyperdata learningcited by 1
  26. 2022On Connecting Deep Trigonometric Networks with Deep Gaussian Processes: Covariance, Expressivity, and Neural Tangent Kernelcited by 1
  27. 2023Variational Elliptical Processescited by 1
  28. 2021Dense Uncertainty Estimation via an Ensemble-based Conditional Latent Variable Modelcited by 0
  29. 2023Amortised Inference in Bayesian Neural Networkscited by 0
  30. 2025PEdger++: Practical Edge Detection via Assembling Cross Informationcited by 0
  31. 2026Amortising Inference and Meta-Learning Priors in Neural Networkscited by 0