# Deep Variational Implicit Processes

arXiv: https://arxiv.org/abs/2206.06720 · HTML: https://mlsanity.com/paper/2206.06720/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 42 references that are in the map.
- First posted on arXiv 14 Jun 2022.

## Builds on (most cited first, 20 of 20)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Deep Neural Networks as Gaussian Processes](https://mlsanity.com/paper/1711.00165/) (2017)
- [Deep Kernel Learning](https://mlsanity.com/paper/1511.02222/) (2015)
- [Gaussian Processes for Big Data](https://mlsanity.com/paper/1309.6835/) (2013)
- [Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks](https://mlsanity.com/paper/1502.05336/) (2015)
- [Doubly Stochastic Variational Inference for Deep Gaussian Processes](https://mlsanity.com/paper/1705.08933/) (2017)
- [Functional Variational Bayesian Neural Networks](https://mlsanity.com/paper/1903.05779/) (2019)
- [Stochastic Variational Deep Kernel Learning](https://mlsanity.com/paper/1611.00336/) (2016)
- [Variational Fourier features for Gaussian processes](https://mlsanity.com/paper/1611.06740/) (2016)
- [A Spectral Approach to Gradient Estimation for Implicit Distributions](https://mlsanity.com/paper/1806.02925/) (2018)
- [Sparse Gaussian Processes with Spherical Harmonic Features](https://mlsanity.com/paper/2006.16649/) (2020)
- [Variational Implicit Processes](https://mlsanity.com/paper/1806.02390/) (2018)
- [Deep Gaussian Processes for Regression using Approximate Expectation Propagation](https://mlsanity.com/paper/1602.04133/) (2016)
- [Black-box α-divergence Minimization](https://mlsanity.com/paper/1511.03243/) (2015)
- [Avoiding pathologies in very deep networks](https://mlsanity.com/paper/1402.5836/) (2014)
- [Random Feature Expansions for Deep Gaussian Processes](https://mlsanity.com/paper/1610.04386/) (2016)
- [Inference in Deep Gaussian Processes using Stochastic Gradient Hamiltonian Monte Carlo](https://mlsanity.com/paper/1806.05490/) (2018)
- [Implicit Posterior Variational Inference for Deep Gaussian Processes](https://mlsanity.com/paper/1910.11998/) (2019)

## Built on by (most cited first, 8 of 8)

- [Variational Linearized Laplace Approximation for Bayesian Deep Learning](https://mlsanity.com/paper/2302.12565/) (2023)
- [Sparse Inducing Points in Deep Gaussian Processes: Enhancing Modeling with Denoising Diffusion Variational Inference](https://mlsanity.com/paper/2407.17033/) (2024)
- [Spectral Convolutional Conditional Neural Processes](https://mlsanity.com/paper/2404.13182/) (2024)
- [Nonparametric Automatic Differentiation Variational Inference with Spline Approximation](https://mlsanity.com/paper/2403.06302/) (2024)
- [Fixed-Mean Gaussian Processes for Post-hoc Bayesian Deep Learning](https://mlsanity.com/paper/2412.04177/) (2024)
- [Deep Transformed Gaussian Processes](https://mlsanity.com/paper/2310.18230/) (2023)
- [Flow-Transformed Implicit Processes for Function-Space Variational Inference](https://mlsanity.com/paper/2606.01954/) (2026)
- [Deep Q-Exponential Processes](https://mlsanity.com/paper/2410.22119/) (2024)

## Abstract (arXiv)

Implicit processes (IPs) are a generalization of Gaussian processes (GPs). IPs may lack a closed-form expression but are easy to sample from. Examples include, among others, Bayesian neural networks or neural samplers. IPs can be used as priors over functions, resulting in flexible models with well-calibrated prediction uncertainty estimates. Methods based on IPs usually carry out function-space approximate inference, which overcomes some of the difficulties of parameter-space approximate inference. Nevertheless, the approximations employed often limit the expressiveness of the final model, resulting, e.g., in a Gaussian predictive distribution, which can be restrictive. We propose here a multi-layer generalization of IPs called the Deep Variational Implicit process (DVIP). This generalization is similar to that of deep GPs over GPs, but it is more flexible due to the use of IPs as the prior distribution over the latent functions. We describe a scalable variational inference algorithm for training DVIP and show that it outperforms previous IP-based methods and also deep GPs. We support these claims via extensive regression and classification experiments. We also evaluate DVIP on large datasets with up to several million data instances to illustrate its good scalability and performance.