# Deep Gaussian Processes for Regression using Approximate Expectation Propagation

arXiv: https://arxiv.org/abs/1602.04133 · HTML: https://mlsanity.com/paper/1602.04133/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Feb 2016.

## Built on by (most cited first, 30 of 39)

- [Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning](https://mlsanity.com/paper/2106.02584/) (2021)
- [Priors in Bayesian Deep Learning: A Review](https://mlsanity.com/paper/2105.06868/) (2021)
- [CiteME: Can Language Models Accurately Cite Scientific Claims?](https://mlsanity.com/paper/2407.12861/) (2024)
- [Deep networks for system identification: a Survey](https://mlsanity.com/paper/2301.12832/) (2023)
- [Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety](https://mlsanity.com/paper/2104.14235/) (2021)
- [Uncertainty Quantification in Machine Learning for Engineering Design and Health Prognostics: A Tutorial](https://mlsanity.com/paper/2305.04933/) (2023)
- [Deep Neural Networks as Point Estimates for Deep Gaussian Processes](https://mlsanity.com/paper/2105.04504/) (2021)
- [The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspective](https://mlsanity.com/paper/2106.06529/) (2021)
- [Deep neural networks with dependent weights: Gaussian Process mixture limit, heavy tails, sparsity and compressibility](https://mlsanity.com/paper/2205.08187/) (2022)
- [Deep Gaussian Process Emulation using Stochastic Imputation](https://mlsanity.com/paper/2107.01590/) (2021)
- [DNNR: Differential Nearest Neighbors Regression](https://mlsanity.com/paper/2205.08434/) (2022)
- [Deep Variational Implicit Processes](https://mlsanity.com/paper/2206.06720/) (2022)
- [Variational Continual Test-Time Adaptation](https://mlsanity.com/paper/2402.08182/) (2024)
- [Collapsed Inference for Bayesian Deep Learning](https://mlsanity.com/paper/2306.09686/) (2023)
- [Gaussian Process Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2407.18397/) (2024)
- [A Sparse Expansion For Deep Gaussian Processes](https://mlsanity.com/paper/2112.05888/) (2021)
- [Revisiting L1 Loss in Super-Resolution: A Probabilistic View and Beyond](https://mlsanity.com/paper/2201.10084/) (2022)
- [Gaussian Processes for Missing Value Imputation](https://mlsanity.com/paper/2204.04648/) (2022)
- [An Anomaly Detection Method for Satellites Using Monte Carlo Dropout](https://mlsanity.com/paper/2211.14938/) (2022)
- [A Flexible Nadaraya-Watson Head Can Offer Explainable and Calibrated Classification](https://mlsanity.com/paper/2212.03411/) (2022)
- [Tighter sparse variational Gaussian processes](https://mlsanity.com/paper/2502.04750/) (2025)
- [Convolutional Normalizing Flows for Deep Gaussian Processes](https://mlsanity.com/paper/2104.08472/) (2021)
- [Modelling Non-Smooth Signals with Complex Spectral Structure](https://mlsanity.com/paper/2203.06997/) (2022)
- [Measuring Uncertainty in Signal Fingerprinting with Gaussian Processes Going Deep](https://mlsanity.com/paper/2109.04360/) (2021)
- [Conditional Deep Gaussian Processes: empirical Bayes hyperdata learning](https://mlsanity.com/paper/2110.00568/) (2021)
- [On Connecting Deep Trigonometric Networks with Deep Gaussian Processes: Covariance, Expressivity, and Neural Tangent Kernel](https://mlsanity.com/paper/2203.07411/) (2022)
- [Shallow and Deep Nonparametric Convolutions for Gaussian Processes](https://mlsanity.com/paper/2206.08972/) (2022)
- [Unlearning in Diffusion models under Data Constraints: A Variational Inference Approach](https://mlsanity.com/paper/2510.04058/) (2025)
- [Knowledge Distillation of Uncertainty using Deep Latent Factor Model](https://mlsanity.com/paper/2510.19290/) (2025)
- [Highly Efficient Representation and Active Learning Framework and Its Application to Imbalanced Medical Image Classification](https://mlsanity.com/paper/2103.05109/) (2021)

## Abstract (arXiv)

Deep Gaussian processes (DGPs) are multi-layer hierarchical generalisations of Gaussian processes (GPs) and are formally equivalent to neural networks with multiple, infinitely wide hidden layers. DGPs are nonparametric probabilistic models and as such are arguably more flexible, have a greater capacity to generalise, and provide better calibrated uncertainty estimates than alternative deep models. This paper develops a new approximate Bayesian learning scheme that enables DGPs to be applied to a range of medium to large scale regression problems for the first time. The new method uses an approximate Expectation Propagation procedure and a novel and efficient extension of the probabilistic backpropagation algorithm for learning. We evaluate the new method for non-linear regression on eleven real-world datasets, showing that it always outperforms GP regression and is almost always better than state-of-the-art deterministic and sampling-based approximate inference methods for Bayesian neural networks. As a by-product, this work provides a comprehensive analysis of six approximate Bayesian methods for training neural networks.