# A Sparse Expansion For Deep Gaussian Processes

arXiv: https://arxiv.org/abs/2112.05888 · HTML: https://mlsanity.com/paper/2112.05888/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 62 references that are in the map.
- First posted on arXiv 11 Dec 2021.

## Builds on (most cited first, 19 of 19)

- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Automatic differentiation in machine learning: a survey](https://mlsanity.com/paper/1502.05767/) (2015)
- [Doubly Stochastic Variational Inference for Deep Gaussian Processes](https://mlsanity.com/paper/1705.08933/) (2017)
- [A general framework for Vecchia approximations of Gaussian processes](https://mlsanity.com/paper/1708.06302/) (2017)
- [Deep Gaussian Processes for Regression using Approximate Expectation Propagation](https://mlsanity.com/paper/1602.04133/) (2016)
- [Active Learning for Deep Gaussian Process Surrogates](https://mlsanity.com/paper/2012.08015/) (2020)
- [Scalable Kernel Methods via Doubly Stochastic Gradients](https://mlsanity.com/paper/1407.5599/) (2014)
- [Random Feature Expansions for Deep Gaussian Processes](https://mlsanity.com/paper/1610.04386/) (2016)
- [Inference in Deep Gaussian Processes using Stochastic Gradient Hamiltonian Monte Carlo](https://mlsanity.com/paper/1806.05490/) (2018)
- [The Variational Gaussian Process](https://mlsanity.com/paper/1511.06499/) (2015)
- [Bayesian Optimization using Deep Gaussian Processes](https://mlsanity.com/paper/1905.03350/) (2019)
- [Variational Auto-encoded Deep Gaussian Processes](https://mlsanity.com/paper/1511.06455/) (2015)
- [Deep Bayesian Gaussian Processes for Uncertainty Estimation in Electronic Health Records](https://mlsanity.com/paper/2003.10170/) (2020)
- [Deep Gaussian Markov Random Fields](https://mlsanity.com/paper/2002.07467/) (2020)
- [Fast prediction of deterministic functions using sparse grid experimental designs](https://mlsanity.com/paper/1402.6350/) (2014)
- [Nested Variational Compression in Deep Gaussian Processes](https://mlsanity.com/paper/1412.1370/) (2014)
- [Generalization Guarantees for Sparse Kernel Approximation with Entropic Optimal Features](https://mlsanity.com/paper/2002.04195/) (2020)
- [Bayesian active learning for choice models with deep Gaussian processes](https://mlsanity.com/paper/1805.01867/) (2018)

## Built on by (most cited first, 5 of 5)

- [From Deep Additive Kernel Learning to Last-Layer Bayesian Neural Networks via Induced Prior Approximation](https://mlsanity.com/paper/2502.10540/) (2025)
- [Privacy-aware Gaussian Process Regression](https://mlsanity.com/paper/2305.16541/) (2023)
- [Gaussian Processes Sampling with Sparse Grids under Additive Schwarz Preconditioner](https://mlsanity.com/paper/2408.00206/) (2024)
- [Aggregation Models with Optimal Weights for Distributed Gaussian Processes](https://mlsanity.com/paper/2408.00955/) (2024)
- [SIKA-GP: Accelerating Gaussian Process Inference with Sparse Inducing Kernel Approximations for Bayesian Deep Learning](https://mlsanity.com/paper/2605.26509/) (2026)

## Abstract (arXiv)

In this work, we use Deep Gaussian Processes (DGPs) as statistical surrogates for stochastic processes with complex distributions. Conventional inferential methods for DGP models can suffer from high computational complexity as they require large-scale operations with kernel matrices for training and inference. In this work, we propose an efficient scheme for accurate inference and efficient training based on a range of Gaussian Processes, called the Tensor Markov Gaussian Processes (TMGP). We construct an induced approximation of TMGP referred to as the hierarchical expansion. Next, we develop a deep TMGP (DTMGP) model as the composition of multiple hierarchical expansion of TMGPs. The proposed DTMGP model has the following properties: (1) the outputs of each activation function are deterministic while the weights are chosen independently from standard Gaussian distribution; (2) in training or prediction, only polylog(M) (out of M) activation functions have non-zero outputs, which significantly boosts the computational efficiency. Our numerical experiments on synthetic models and real datasets show the superior computational efficiency of DTMGP over existing DGP models.