# Fast and Scalable Spike and Slab Variable Selection in High-Dimensional Gaussian Processes

arXiv: https://arxiv.org/abs/2111.04558 · HTML: https://mlsanity.com/paper/2111.04558/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 54 references that are in the map.
- First posted on arXiv 8 Nov 2021.

## Builds on (most cited first, 25 of 25)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration](https://mlsanity.com/paper/1809.11165/) (2018)
- [Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks](https://mlsanity.com/paper/2102.00554/) (2021)
- [Deep Kernel Learning](https://mlsanity.com/paper/1511.02222/) (2015)
- [Gaussian Processes for Big Data](https://mlsanity.com/paper/1309.6835/) (2013)
- [Black Box Variational Inference](https://mlsanity.com/paper/1401.0118/) (2013)
- [Monte Carlo Gradient Estimation in Machine Learning](https://mlsanity.com/paper/1906.10652/) (2019)
- [On the Origin of Implicit Regularization in Stochastic Gradient Descent](https://mlsanity.com/paper/2101.12176/) (2021)
- [Exact Gaussian Processes on a Million Data Points](https://mlsanity.com/paper/1903.08114/) (2019)
- [Bayesian Compression for Deep Learning](https://mlsanity.com/paper/1705.08665/) (2017)
- [The Promises and Pitfalls of Deep Kernel Learning](https://mlsanity.com/paper/2102.12108/) (2021)
- [Rates of Convergence for Sparse Variational Gaussian Process Regression](https://mlsanity.com/paper/1903.03571/) (2019)
- [Understanding predictive information criteria for Bayesian models](https://mlsanity.com/paper/1307.5928/) (2013)
- [MCMC for Variationally Sparse Gaussian Processes](https://mlsanity.com/paper/1506.04000/) (2015)
- [Bayesian linear regression with sparse priors](https://mlsanity.com/paper/1403.0735/) (2014)
- [Using stacking to average Bayesian predictive distributions](https://mlsanity.com/paper/1704.02030/) (2017)
- [Rao-Blackwellized Stochastic Gradients for Discrete Distributions](https://mlsanity.com/paper/1810.04777/) (2018)
- [Variable selection for Gaussian processes via sensitivity analysis of the posterior predictive distribution](https://mlsanity.com/paper/1712.08048/) (2017)
- [Spike-and-Slab Meets LASSO: A Review of the Spike-and-Slab LASSO](https://mlsanity.com/paper/2010.06451/) (2020)
- [Scalable Cross Validation Losses for Gaussian Process Models](https://mlsanity.com/paper/2105.11535/) (2021)
- [Additive Gaussian Process Regression](https://mlsanity.com/paper/1411.7009/) (2014)
- [Efficient leave-one-out cross-validation for Bayesian non-factorized normal and Student-t models](https://mlsanity.com/paper/1810.10559/) (2018)
- [Spike and Slab Gaussian Process Latent Variable Models](https://mlsanity.com/paper/1505.02434/) (2015)

## Built on by (most cited first, 7 of 7)

- [Non-Stationary Learning of Neural Networks with Automatic Soft Parameter Reset](https://mlsanity.com/paper/2411.04034/) (2024)
- [Interventional Processes for Causal Uncertainty Quantification](https://mlsanity.com/paper/2410.14483/) (2024)
- [Non-stationary and Sparsely-correlated Multi-output Gaussian Process with Spike-and-Slab Prior](https://mlsanity.com/paper/2409.03149/) (2024)
- [RFFNet: Large-Scale Interpretable Kernel Methods via Random Fourier Features](https://mlsanity.com/paper/2211.06410/) (2022)
- [Nowcasting with Mixed Frequency Data Using Gaussian Processes](https://mlsanity.com/paper/2402.10574/) (2024)
- [Variational Task Vector Composition](https://mlsanity.com/paper/2509.18208/) (2025)
- [Sparsity by Default: The Theory and Practice of ARD in Gaussian Process Regression for Variable Selection](https://mlsanity.com/paper/2609.33550/) (2026)

## Abstract (arXiv)

Variable selection in Gaussian processes (GPs) is typically undertaken by thresholding the inverse lengthscales of automatic relevance determination kernels, but in high-dimensional datasets this approach can be unreliable. A more probabilistically principled alternative is to use spike and slab priors and infer a posterior probability of variable inclusion. However, existing implementations in GPs are very costly to run in both high-dimensional and large-n datasets, or are only suitable for unsupervised settings with specific kernels. As such, we develop a fast and scalable variational inference algorithm for the spike and slab GP that is tractable with arbitrary differentiable kernels. We improve our algorithm's ability to adapt to the sparsity of relevant variables by Bayesian model averaging over hyperparameters, and achieve substantial speed ups using zero temperature posterior restrictions, dropout pruning and nearest neighbour minibatching. In experiments our method consistently outperforms vanilla and sparse variational GPs whilst retaining similar runtimes (even when n=106) and performs competitively with a spike and slab GP using MCMC but runs up to 1000 times faster.