# Inference in Deep Gaussian Processes using Stochastic Gradient Hamiltonian Monte Carlo

arXiv: https://arxiv.org/abs/1806.05490 · HTML: https://mlsanity.com/paper/1806.05490/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jun 2018.

## Built on by (most cited first, 30 of 32)

- [Causal Inference for Time series Analysis: Problems, Methods and Evaluation](https://mlsanity.com/paper/2102.05829/) (2021)
- [Uncertainty Quantification in Machine Learning for Engineering Design and Health Prognostics: A Tutorial](https://mlsanity.com/paper/2305.04933/) (2023)
- [The Neural Process Family: Survey, Applications and Perspectives](https://mlsanity.com/paper/2209.00517/) (2022)
- [The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspective](https://mlsanity.com/paper/2106.06529/) (2021)
- [Bayesian Interpolation with Deep Linear Networks](https://mlsanity.com/paper/2212.14457/) (2022)
- [Sparse Inducing Points in Deep Gaussian Processes: Enhancing Modeling with Denoising Diffusion Variational Inference](https://mlsanity.com/paper/2407.17033/) (2024)
- [GPflux: A Library for Deep Gaussian Processes](https://mlsanity.com/paper/2104.05674/) (2021)
- [Deep Gaussian Processes: A Survey](https://mlsanity.com/paper/2106.12135/) (2021)
- [Deep Gaussian Process Emulation using Stochastic Imputation](https://mlsanity.com/paper/2107.01590/) (2021)
- [Universal Functional Regression with Neural Operator Flows](https://mlsanity.com/paper/2404.02986/) (2024)
- [Compositional Modeling of Nonlinear Dynamical Systems with ODE-based Random Features](https://mlsanity.com/paper/2106.05960/) (2021)
- [Deep Variational Implicit Processes](https://mlsanity.com/paper/2206.06720/) (2022)
- [A Unifying Perspective on Non-Stationary Kernels for Deeper Gaussian Processes](https://mlsanity.com/paper/2309.10068/) (2023)
- [Free-Form Variational Inference for Gaussian Process State-Space Models](https://mlsanity.com/paper/2302.09921/) (2023)
- [Neural Operator Variational Inference based on Regularized Stein Discrepancy for Deep Gaussian Processes](https://mlsanity.com/paper/2309.12658/) (2023)
- [A Sparse Expansion For Deep Gaussian Processes](https://mlsanity.com/paper/2112.05888/) (2021)
- [Sparse Gaussian Process Hyperparameters: Optimize or Integrate?](https://mlsanity.com/paper/2211.02476/) (2022)
- [Fully Bayesian Differential Gaussian Processes through Stochastic Differential Equations](https://mlsanity.com/paper/2408.06069/) (2024)
- [Diffusion Bridge Variational Inference for Deep Gaussian Processes](https://mlsanity.com/paper/2509.19078/) (2025)
- [Convolutional Normalizing Flows for Deep Gaussian Processes](https://mlsanity.com/paper/2104.08472/) (2021)
- [Probabilistic Metamodels for an Efficient Characterization of Complex Driving Scenarios](https://mlsanity.com/paper/2110.02892/) (2021)
- [Deep Gaussian Process-based Cost-Aware Batch Bayesian Optimization for Complex Materials Design Campaigns](https://mlsanity.com/paper/2509.14408/) (2025)
- [Deep Jump Gaussian Processes for Surrogate Modeling of High-Dimensional Piecewise Continuous Functions](https://mlsanity.com/paper/2510.21974/) (2025)
- [Conditional Deep Gaussian Processes: empirical Bayes hyperdata learning](https://mlsanity.com/paper/2110.00568/) (2021)
- [Distributional Gaussian Processes Layers for Out-of-Distribution Detection](https://mlsanity.com/paper/2206.13346/) (2022)
- [Deep Transformed Gaussian Processes](https://mlsanity.com/paper/2310.18230/) (2023)
- [Deep Latent Force Models: ODE-based Process Convolutions for Bayesian Deep Learning](https://mlsanity.com/paper/2311.14828/) (2023)
- [Adaptive RKHS Fourier Features for Compositional Gaussian Process Models](https://mlsanity.com/paper/2407.01856/) (2024)
- [Onsager-Machlup Posterior Transport for Deep Gaussian Processes](https://mlsanity.com/paper/2605.23434/) (2026)
- [Subset-of-Data Variational Inference for Deep Gaussian-Processes Regression](https://mlsanity.com/paper/2107.08265/) (2021)

## Abstract (arXiv)

Deep Gaussian Processes (DGPs) are hierarchical generalizations of Gaussian Processes that combine well calibrated uncertainty estimates with the high flexibility of multilayer models. One of the biggest challenges with these models is that exact inference is intractable. The current state-of-the-art inference method, Variational Inference (VI), employs a Gaussian approximation to the posterior distribution. This can be a potentially poor unimodal approximation of the generally multimodal posterior. In this work, we provide evidence for the non-Gaussian nature of the posterior and we apply the Stochastic Gradient Hamiltonian Monte Carlo method to generate samples. To efficiently optimize the hyperparameters, we introduce the Moving Window MCEM algorithm. This results in significantly better predictions at a lower computational cost than its VI counterpart. Thus our method establishes a new state-of-the-art for inference in DGPs.