# Function-space Inference with Sparse Implicit Processes

arXiv: https://arxiv.org/abs/2110.07618 · HTML: https://mlsanity.com/paper/2110.07618/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 39 references that are in the map.
- First posted on arXiv 14 Oct 2021.

## Builds on (most cited first, 13 of 13)

- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm](https://mlsanity.com/paper/1608.04471/) (2016)
- [Gaussian Processes for Big Data](https://mlsanity.com/paper/1309.6835/) (2013)
- [Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks](https://mlsanity.com/paper/1502.05336/) (2015)
- [Functional Variational Bayesian Neural Networks](https://mlsanity.com/paper/1903.05779/) (2019)
- [Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks](https://mlsanity.com/paper/1701.04722/) (2017)
- [Generalized Variational Inference: Three arguments for deriving new Posteriors](https://mlsanity.com/paper/1904.02063/) (2019)
- [Learning and Policy Search in Stochastic Dynamical Systems with Bayesian Neural Networks](https://mlsanity.com/paper/1605.07127/) (2016)
- [Variational Implicit Processes](https://mlsanity.com/paper/1806.02390/) (2018)
- [Understanding Variational Inference in Function-Space](https://mlsanity.com/paper/2011.09421/) (2020)
- [Sparse Uncertainty Representation in Deep Learning with Inducing Weights](https://mlsanity.com/paper/2105.14594/) (2021)
- [Wide Mean-Field Variational Bayesian Neural Networks Ignore the Data](https://mlsanity.com/paper/2106.07052/) (2021)

## Built on by (most cited first, 13 of 13)

- [Do Bayesian Neural Networks Need To Be Fully Stochastic?](https://mlsanity.com/paper/2211.06291/) (2022)
- [A Rigorous Link between Deep Ensembles and (Variational) Bayesian Methods](https://mlsanity.com/paper/2305.15027/) (2023)
- [Neural Operator Variational Inference based on Regularized Stein Discrepancy for Deep Gaussian Processes](https://mlsanity.com/paper/2309.12658/) (2023)
- [Variational Mixture of HyperGenerators for Learning Distributions Over Functions](https://mlsanity.com/paper/2302.06223/) (2023)
- [Spectral Convolutional Conditional Neural Processes](https://mlsanity.com/paper/2404.13182/) (2024)
- [FedSI: Federated Subnetwork Inference for Efficient Uncertainty Quantification](https://mlsanity.com/paper/2404.15657/) (2024)
- [Adaptive Conditional Quantile Neural Processes](https://mlsanity.com/paper/2305.18777/) (2023)
- [Training Implicit Generative Models via an Invariant Statistical Loss](https://mlsanity.com/paper/2402.16435/) (2024)
- [Nonparametric Automatic Differentiation Variational Inference with Spline Approximation](https://mlsanity.com/paper/2403.06302/) (2024)
- [Function Space Bayesian Pseudocoreset for Bayesian Neural Networks](https://mlsanity.com/paper/2310.17852/) (2023)
- [Correcting Model Bias with Sparse Implicit Processes](https://mlsanity.com/paper/2207.10673/) (2022)
- [R'enyi Neural Processes](https://mlsanity.com/paper/2405.15991/) (2024)
- [Flexible Bayesian Last Layer Models Using Implicit Priors and Diffusion Posterior Sampling](https://mlsanity.com/paper/2408.03746/) (2024)

## Abstract (arXiv)

Implicit Processes (IPs) represent a flexible framework that can be used to describe a wide variety of models, from Bayesian neural networks, neural samplers and data generators to many others. IPs also allow for approximate inference in function-space. This change of formulation solves intrinsic degenerate problems of parameter-space approximate inference concerning the high number of parameters and their strong dependencies in large models. For this, previous works in the literature have attempted to employ IPs both to set up the prior and to approximate the resulting posterior. However, this has proven to be a challenging task. Existing methods that can tune the prior IP result in a Gaussian predictive distribution, which fails to capture important data patterns. By contrast, methods producing flexible predictive distributions by using another IP to approximate the posterior process cannot tune the prior IP to the observed data. We propose here the first method that can accomplish both goals. For this, we rely on an inducing-point representation of the prior IP, as often done in the context of sparse Gaussian processes. The result is a scalable method for approximate inference with IPs that can tune the prior IP parameters to the data, and that provides accurate non-Gaussian predictive distributions.