# Deep Horseshoe Gaussian Processes

arXiv: https://arxiv.org/abs/2403.01737 · HTML: https://mlsanity.com/paper/2403.01737/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 59 references that are in the map.
- First posted on arXiv 4 Mar 2024.

## Builds on (most cited first, 16 of 16)

- [Nonparametric regression using deep neural networks with ReLU activation function](https://mlsanity.com/paper/1708.06633/) (2017)
- [Gaussian Process Behaviour in Wide Deep Neural Networks](https://mlsanity.com/paper/1804.11271/) (2018)
- [Doubly Stochastic Variational Inference for Deep Gaussian Processes](https://mlsanity.com/paper/1705.08933/) (2017)
- [Posterior Concentration for Sparse Deep Learning](https://mlsanity.com/paper/1803.09138/) (2018)
- [Efficient Variational Inference for Sparse Deep Learning with Theoretical Guarantee](https://mlsanity.com/paper/2011.07439/) (2020)
- [Steps Toward Deep Kernel Methods from Infinite Neural Networks](https://mlsanity.com/paper/1508.05133/) (2015)
- [Convergence Rates of Variational Inference in Sparse Deep Learning](https://mlsanity.com/paper/1908.04847/) (2019)
- [Deep Gaussian Processes with Importance-Weighted Variational Inference](https://mlsanity.com/paper/1905.05435/) (2019)
- [Frequentist coverage and sup-norm convergence rate in Gaussian process regression](https://mlsanity.com/paper/1708.04753/) (2017)
- [Adaptive variational Bayes: Optimality, computation and applications](https://mlsanity.com/paper/2109.03204/) (2021)
- [GPflux: A Library for Deep Gaussian Processes](https://mlsanity.com/paper/2104.05674/) (2021)
- [Deep State-Space Gaussian Processes](https://mlsanity.com/paper/2008.04733/) (2020)
- [On the inability of Gaussian process regression to optimally learn compositional functions](https://mlsanity.com/paper/2205.07764/) (2022)
- [Variable Selection Consistency of Gaussian Process Regression](https://mlsanity.com/paper/1912.05738/) (2019)
- [Semiparametric inference using fractional posteriors](https://mlsanity.com/paper/2301.08158/) (2023)
- [Non-stationary Gaussian process discriminant analysis with variable selection for high-dimensional functional data](https://mlsanity.com/paper/2109.14171/) (2021)

## Built on by (most cited first, 5 of 5)

- [Posterior and variational inference for deep neural networks with heavy-tailed weights](https://mlsanity.com/paper/2406.03369/) (2024)
- [Minimax optimality of deep neural networks on dependent data via PAC-Bayes bounds](https://mlsanity.com/paper/2410.21702/) (2024)
- [Adaptive Resolution for Finite-Rank Gaussian Processes](https://mlsanity.com/paper/2505.24066/) (2025)
- [Deep Q-Exponential Processes](https://mlsanity.com/paper/2410.22119/) (2024)
- [Sparsity by Default: The Theory and Practice of ARD in Gaussian Process Regression for Variable Selection](https://mlsanity.com/paper/2609.33550/) (2026)

## Abstract (arXiv)

Deep Gaussian processes have recently been proposed as natural objects to fit, similarly to deep neural networks, possibly complex features present in modern data samples, such as compositional structures. Adopting a Bayesian nonparametric approach, it is natural to use deep Gaussian processes as prior distributions, and use the corresponding posterior distributions for statistical inference. We introduce the deep Horseshoe Gaussian process Deep-HGP, a new simple prior based on deep Gaussian processes with a squared-exponential kernel, that in particular enables data-driven choices of the key lengthscale parameters. For nonparametric regression with random design, we show that the associated posterior distribution recovers the unknown true regression curve optimally in terms of quadratic loss, up to a logarithmic factor, in an adaptive way. The convergence rates are simultaneously adaptive to both the smoothness of the regression function and to its structure in terms of compositions. The dependence of the rates in terms of dimension are explicit, allowing in particular for input spaces of dimension increasing with the number of observations.