# Doubly Stochastic Variational Inference for Deep Gaussian Processes

arXiv: https://arxiv.org/abs/1705.08933 · HTML: https://mlsanity.com/paper/1705.08933/

## Key facts

- Cited by 143 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 May 2017.

## Built on by (most cited first, 30 of 143)

- [Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning](https://mlsanity.com/paper/2106.02584/) (2021)
- [Priors in Bayesian Deep Learning: A Review](https://mlsanity.com/paper/2105.06868/) (2021)
- [The Promises and Pitfalls of Deep Kernel Learning](https://mlsanity.com/paper/2102.12108/) (2021)
- [A Simple Approach to Improve Single-Model Deep Uncertainty via Distance-Awareness](https://mlsanity.com/paper/2205.00403/) (2022)
- [Safe Exploration in Reinforcement Learning: A Generalized Formulation and Algorithms](https://mlsanity.com/paper/2310.03225/) (2023)
- [Uncertainty Quantification in Machine Learning for Engineering Design and Health Prognostics: A Tutorial](https://mlsanity.com/paper/2305.04933/) (2023)
- [Deep Neural Networks as Point Estimates for Deep Gaussian Processes](https://mlsanity.com/paper/2105.04504/) (2021)
- [The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspective](https://mlsanity.com/paper/2106.06529/) (2021)
- [Calibrating Transformers via Sparse Gaussian Processes](https://mlsanity.com/paper/2303.02444/) (2023)
- [Variational Linearized Laplace Approximation for Bayesian Deep Learning](https://mlsanity.com/paper/2302.12565/) (2023)
- [Adaptive Kernel Design for Bayesian Optimization Is a Piece of CAKE with LLMs](https://mlsanity.com/paper/2509.17998/) (2025)
- [Sparse Uncertainty Representation in Deep Learning with Inducing Weights](https://mlsanity.com/paper/2105.14594/) (2021)
- [Sparse Inducing Points in Deep Gaussian Processes: Enhancing Modeling with Denoising Diffusion Variational Inference](https://mlsanity.com/paper/2407.17033/) (2024)
- [How to Evaluate Uncertainty Estimates in Machine Learning for Regression?](https://mlsanity.com/paper/2106.03395/) (2021)
- [A theory of representation learning gives a deep generalisation of kernel methods](https://mlsanity.com/paper/2108.13097/) (2021)
- [Statistical Deep Learning for Spatial and Spatio-Temporal Data](https://mlsanity.com/paper/2206.02218/) (2022)
- [Multi-fidelity Hierarchical Neural Processes](https://mlsanity.com/paper/2206.04872/) (2022)
- [Stochastic Process Learning via Operator Flow Matching](https://mlsanity.com/paper/2501.04126/) (2025)
- [GPflux: A Library for Deep Gaussian Processes](https://mlsanity.com/paper/2104.05674/) (2021)
- [Deep Gaussian Processes: A Survey](https://mlsanity.com/paper/2106.12135/) (2021)
- [Multi-Fidelity Residual Neural Processes for Scalable Surrogate Modeling](https://mlsanity.com/paper/2402.18846/) (2024)
- [Deep Gaussian Process Emulation using Stochastic Imputation](https://mlsanity.com/paper/2107.01590/) (2021)
- [Triangulation candidates for Bayesian optimization](https://mlsanity.com/paper/2112.07457/) (2021)
- [Trieste: Efficiently Exploring The Depths of Black-box Functions with TensorFlow](https://mlsanity.com/paper/2302.08436/) (2023)
- [Stopping Bayesian Optimization with Probabilistic Regret Bounds](https://mlsanity.com/paper/2402.16811/) (2024)
- [SKIing on Simplices: Kernel Interpolation on the Permutohedral Lattice for Scalable Gaussian Processes](https://mlsanity.com/paper/2106.06695/) (2021)
- [Fully Bayesian Autoencoders with Latent Sparse Gaussian Processes](https://mlsanity.com/paper/2302.04534/) (2023)
- [Universal Functional Regression with Neural Operator Flows](https://mlsanity.com/paper/2404.02986/) (2024)
- [Residual Gaussian Process: A Tractable Nonparametric Bayesian Emulator for Multi-fidelity Simulations](https://mlsanity.com/paper/2104.03743/) (2021)
- [Compositional Modeling of Nonlinear Dynamical Systems with ODE-based Random Features](https://mlsanity.com/paper/2106.05960/) (2021)

## Abstract (arXiv)

Gaussian processes (GPs) are a good choice for function approximation as they are flexible, robust to over-fitting, and provide well-calibrated predictive uncertainty. Deep Gaussian processes (DGPs) are multi-layer generalisations of GPs, but inference in these models has proved challenging. Existing approaches to inference in DGP models assume approximate posteriors that force independence between the layers, and do not work well in practice. We present a doubly stochastic variational inference algorithm, which does not force independence between layers. With our method of inference we demonstrate that a DGP model can be used effectively on data ranging in size from hundreds to a billion points. We provide strong empirical evidence that our inference scheme for DGPs works well in practice in both classification and regression.