mlsanityData to 14 Oct 2026

Sparsity by Default: The Theory and Practice of ARD in Gaussian Process Regression for Variable Selection

arXiv 2609.33550 ↗First posted 27 Sep 2026stat.ML cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Automatic relevance determination (ARD) is the standard device for input selection in Gaussian process (GP) regression. By giving the covariance kernel a separate lengthscale for every input and learning those lengthscales by maximizing the marginal likelihood, ARD lets the data decide which coordinates matter: irrelevant inputs receive very large lengthscales and are effectively switched off. We trace this mechanism to the Bayesian Occam's razor embodied in the marginal likelihood, derive the gradient through which it prunes inputs, and emphasize that ARD delivers effective rather than exact sparsity. We review the algorithms used in practice and the rules that turn lengthscales into selections, and we survey the asymptotic theory, distinguishing the fixed-domain identifiability obstruction on the lengthscales from the high-dimensional selection-consistency guarantees recently established for hierarchical GP priors, and noting what remains open for plain ARD. We compare ARD with spike-and-slab priors, sparse axis-aligned and global-local shrinkage priors including the Bayesian lasso and horseshoe, penalized likelihood kriging, sensitivity and projection criteria, and additive kernels. We argue that ARD endures because of its seamless integration with kernel learning, universal software support, and low cost, and we close with its limitations and remedies.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 28 of its 87 references that are in the map. First posted on arXiv 27 Sep 2026.

Builds on

28 of 87 references.

References · 28

  1. 2018GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Accelerationcited by 609
  2. 2013Gaussian Processes for Big Datacited by 383
  3. 2014Scalable Variational Gaussian Process Classificationcited by 228
  4. 2018When Gaussian Process Meets Big Data: A Review of Scalable GPscited by 206
  5. 2015Kernel Interpolation for Scalable Structured Gaussian Processes (KISS-GP)cited by 177
  6. 2021High-Dimensional Bayesian Optimization with Sparse Axis-Aligned Subspacescited by 141
  7. 2016GPflow: A Gaussian process library using TensorFlowcited by 140
  8. 2016Panning for Gold: Model-X Knockoffs for High-dimensional Controlled Variable Selectioncited by 121
  9. 2013Structure Discovery in Nonparametric Regression through Compositional Kernel Searchcited by 88
  10. 2014Hierarchical Nearest-Neighbor Gaussian Process Models for Large Geostatistical Datasetscited by 72
  11. 2017Sparsity information and regularization in the horseshoe and other shrinkage priorscited by 43
  12. 2014Bayesian linear regression with sparse priorscited by 40
16 more
  1. 2020Active Learning for Deep Gaussian Process Surrogatescited by 38
  2. 2017Lasso Meets Horseshoe : A Surveycited by 17
  3. 2017Variable selection for Gaussian processes via sensitivity analysis of the posterior predictive distributioncited by 15
  4. 2022Scalable Gaussian-process regression and variable selection using Vecchia approximationscited by 8
  5. 2021Fast and Scalable Spike and Slab Variable Selection in High-Dimensional Gaussian Processescited by 7
  6. 2019Variable Selection Consistency of Gaussian Process Regressioncited by 6
  7. 2023Active Learning of Piecewise Gaussian Process Surrogatescited by 5
  8. 2024Deep Horseshoe Gaussian Processescited by 5
  9. 2015Projection predictive model selection for Gaussian processescited by 3
  10. 2019A Bayesian Lasso based Sparse Learning Modelcited by 3
  11. 2021Kernel Knockoffs Selection for Nonparametric Additive Modelscited by 3
  12. 2018Jointly Robust Prior for Gaussian Stochastic Process in Emulation, Calibration and Variable Selectioncited by 2
  13. 2020Variable selection for Gaussian process regression through a sparse projectioncited by 2
  14. 2025Deep Jump Gaussian Processes for Surrogate Modeling of High-Dimensional Piecewise Continuous Functionscited by 2
  15. 2021Numerical Characterization of Support Recovery in Sparse Regression with Correlated Designcited by 1
  16. 2025Bayesian Bridge Gaussian Process Regressioncited by 1

Built on by

Not cited yet.