# Understanding Variational Inference in Function-Space

arXiv: https://arxiv.org/abs/2011.09421 · HTML: https://mlsanity.com/paper/2011.09421/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Nov 2020.

## Built on by (most cited first, 30 of 40)

- [Repulsive Deep Ensembles are Bayesian](https://mlsanity.com/paper/2106.11642/) (2021)
- [Priors in Bayesian Deep Learning: A Review](https://mlsanity.com/paper/2105.06868/) (2021)
- [Pre-trained Gaussian Processes for Bayesian Optimization](https://mlsanity.com/paper/2109.08215/) (2021)
- [Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AI](https://mlsanity.com/paper/2402.00809/) (2024)
- [Diffusion Generative Models in Infinite Dimensions](https://mlsanity.com/paper/2212.00886/) (2022)
- [Tractable Function-Space Variational Inference in Bayesian Neural Networks](https://mlsanity.com/paper/2312.17199/) (2023)
- [Continual Learning via Sequential Function-Space Variational Inference](https://mlsanity.com/paper/2312.17210/) (2023)
- [Meta-Learning Reliable Priors in the Function Space](https://mlsanity.com/paper/2106.03195/) (2021)
- [On Stein Variational Neural Network Ensembles](https://mlsanity.com/paper/2106.10760/) (2021)
- [Generalized Variational Inference in Function Spaces: Gaussian Measures meet Bayesian Deep Learning](https://mlsanity.com/paper/2205.06342/) (2022)
- [Function-Space Regularization in Neural Networks: A Probabilistic Perspective](https://mlsanity.com/paper/2312.17162/) (2023)
- [Position: The Future of Bayesian Prediction Is Prior-Fitted](https://mlsanity.com/paper/2505.23947/) (2025)
- [Function-space Inference with Sparse Implicit Processes](https://mlsanity.com/paper/2110.07618/) (2021)
- [Globally injective and bijective neural operators](https://mlsanity.com/paper/2306.03982/) (2023)
- [FSP-Laplace: Function-Space Priors for the Laplace Approximation in Bayesian Deep Learning](https://mlsanity.com/paper/2407.13711/) (2024)
- [Incorporating Unlabelled Data into Bayesian Neural Networks](https://mlsanity.com/paper/2304.01762/) (2023)
- [On the detrimental effect of invariances in the likelihood for variational inference](https://mlsanity.com/paper/2209.07157/) (2022)
- [Random Grid Neural Processes for Parametric Partial Differential Equations](https://mlsanity.com/paper/2301.11040/) (2023)
- [Bayes' Power for Explaining In-Context Learning Generalizations](https://mlsanity.com/paper/2410.01565/) (2024)
- [Function Space Diversity for Uncertainty Prediction via Repulsive Last-Layer Ensembles](https://mlsanity.com/paper/2412.15758/) (2024)
- [A variational approximate posterior for the deep Wishart process](https://mlsanity.com/paper/2107.10125/) (2021)
- [On out-of-distribution detection with Bayesian neural networks](https://mlsanity.com/paper/2110.06020/) (2021)
- [Kullback-Leibler and Renyi divergences in reproducing kernel Hilbert space and Gaussian process settings](https://mlsanity.com/paper/2207.08406/) (2022)
- [Spectral Convolutional Conditional Neural Processes](https://mlsanity.com/paper/2404.13182/) (2024)
- [On Uncertainty Quantification for Near-Bayes Optimal Algorithms](https://mlsanity.com/paper/2403.19381/) (2024)
- [Regularized KL-Divergence for Well-Defined Function-Space Variational Inference in Bayesian neural networks](https://mlsanity.com/paper/2406.04317/) (2024)
- [Function Space Bayesian Pseudocoreset for Bayesian Neural Networks](https://mlsanity.com/paper/2310.17852/) (2023)
- [Functional Stochastic Gradient MCMC for Bayesian Neural Networks](https://mlsanity.com/paper/2409.16632/) (2024)
- [Revisiting the Equivalence of Bayesian Neural Networks and Gaussian Processes: On the Importance of Learning Activations](https://mlsanity.com/paper/2410.15777/) (2024)
- [Deep Ensemble as a Gaussian Process Approximate Posterior](https://mlsanity.com/paper/2205.00163/) (2022)

## Abstract (arXiv)

Recent work has attempted to directly approximate the `function-space' or predictive posterior distribution of Bayesian models, without approximating the posterior distribution over the parameters. This is appealing in e.g. Bayesian neural networks, where we only need the former, and the latter is hard to represent. In this work, we highlight some advantages and limitations of employing the Kullback-Leibler divergence in this setting. For example, we show that minimizing the KL divergence between a wide class of parametric distributions and the posterior induced by a (non-degenerate) Gaussian process prior leads to an ill-defined objective function. Then, we propose (featurized) Bayesian linear regression as a benchmark for `function-space' inference methods that directly measures approximation quality. We apply this methodology to assess aspects of the objective function and inference scheme considered in Sun, Zhang, Shi, and Grosse (2018), emphasizing the quality of approximation to Bayesian inference as opposed to predictive performance.