# Sparsity by Default: The Theory and Practice of ARD in Gaussian Process Regression for Variable Selection

arXiv: https://arxiv.org/abs/2609.33550 · HTML: https://mlsanity.com/paper/2609.33550/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 87 references that are in the map.
- First posted on arXiv 27 Sep 2026.

## Builds on (most cited first, 28 of 28)

- [GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration](https://mlsanity.com/paper/1809.11165/) (2018)
- [Gaussian Processes for Big Data](https://mlsanity.com/paper/1309.6835/) (2013)
- [Scalable Variational Gaussian Process Classification](https://mlsanity.com/paper/1411.2005/) (2014)
- [When Gaussian Process Meets Big Data: A Review of Scalable GPs](https://mlsanity.com/paper/1807.01065/) (2018)
- [Kernel Interpolation for Scalable Structured Gaussian Processes (KISS-GP)](https://mlsanity.com/paper/1503.01057/) (2015)
- [High-Dimensional Bayesian Optimization with Sparse Axis-Aligned Subspaces](https://mlsanity.com/paper/2103.00349/) (2021)
- [GPflow: A Gaussian process library using TensorFlow](https://mlsanity.com/paper/1610.08733/) (2016)
- [Panning for Gold: Model-X Knockoffs for High-dimensional Controlled Variable Selection](https://mlsanity.com/paper/1610.02351/) (2016)
- [Structure Discovery in Nonparametric Regression through Compositional Kernel Search](https://mlsanity.com/paper/1302.4922/) (2013)
- [Hierarchical Nearest-Neighbor Gaussian Process Models for Large Geostatistical Datasets](https://mlsanity.com/paper/1406.7343/) (2014)
- [Sparsity information and regularization in the horseshoe and other shrinkage priors](https://mlsanity.com/paper/1707.01694/) (2017)
- [Bayesian linear regression with sparse priors](https://mlsanity.com/paper/1403.0735/) (2014)
- [Active Learning for Deep Gaussian Process Surrogates](https://mlsanity.com/paper/2012.08015/) (2020)
- [Lasso Meets Horseshoe : A Survey](https://mlsanity.com/paper/1706.10179/) (2017)
- [Variable selection for Gaussian processes via sensitivity analysis of the posterior predictive distribution](https://mlsanity.com/paper/1712.08048/) (2017)
- [Scalable Gaussian-process regression and variable selection using Vecchia approximations](https://mlsanity.com/paper/2202.12981/) (2022)
- [Fast and Scalable Spike and Slab Variable Selection in High-Dimensional Gaussian Processes](https://mlsanity.com/paper/2111.04558/) (2021)
- [Variable Selection Consistency of Gaussian Process Regression](https://mlsanity.com/paper/1912.05738/) (2019)
- [Active Learning of Piecewise Gaussian Process Surrogates](https://mlsanity.com/paper/2301.08789/) (2023)
- [Deep Horseshoe Gaussian Processes](https://mlsanity.com/paper/2403.01737/) (2024)
- [Projection predictive model selection for Gaussian processes](https://mlsanity.com/paper/1510.04813/) (2015)
- [A Bayesian Lasso based Sparse Learning Model](https://mlsanity.com/paper/1908.07220/) (2019)
- [Kernel Knockoffs Selection for Nonparametric Additive Models](https://mlsanity.com/paper/2105.11659/) (2021)
- [Jointly Robust Prior for Gaussian Stochastic Process in Emulation, Calibration and Variable Selection](https://mlsanity.com/paper/1804.09329/) (2018)
- [Variable selection for Gaussian process regression through a sparse projection](https://mlsanity.com/paper/2008.10769/) (2020)
- [Deep Jump Gaussian Processes for Surrogate Modeling of High-Dimensional Piecewise Continuous Functions](https://mlsanity.com/paper/2510.21974/) (2025)
- [Numerical Characterization of Support Recovery in Sparse Regression with Correlated Design](https://mlsanity.com/paper/2103.12802/) (2021)
- [Bayesian Bridge Gaussian Process Regression](https://mlsanity.com/paper/2511.17415/) (2025)

## Abstract (arXiv)

Automatic relevance determination (ARD) is the standard device for input selection in Gaussian process (GP) regression. By giving the covariance kernel a separate lengthscale for every input and learning those lengthscales by maximizing the marginal likelihood, ARD lets the data decide which coordinates matter: irrelevant inputs receive very large lengthscales and are effectively switched off. We trace this mechanism to the Bayesian Occam's razor embodied in the marginal likelihood, derive the gradient through which it prunes inputs, and emphasize that ARD delivers effective rather than exact sparsity. We review the algorithms used in practice and the rules that turn lengthscales into selections, and we survey the asymptotic theory, distinguishing the fixed-domain identifiability obstruction on the lengthscales from the high-dimensional selection-consistency guarantees recently established for hierarchical GP priors, and noting what remains open for plain ARD. We compare ARD with spike-and-slab priors, sparse axis-aligned and global-local shrinkage priors including the Bayesian lasso and horseshoe, penalized likelihood kriging, sensitivity and projection criteria, and additive kernels. We argue that ARD endures because of its seamless integration with kernel learning, universal software support, and low cost, and we close with its limitations and remedies.