# Scalable Gaussian-process regression and variable selection using Vecchia approximations

arXiv: https://arxiv.org/abs/2202.12981 · HTML: https://mlsanity.com/paper/2202.12981/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 54 references that are in the map.
- First posted on arXiv 25 Feb 2022.

## Builds on (most cited first, 23 of 23)

- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration](https://mlsanity.com/paper/1809.11165/) (2018)
- [Deep Kernel Learning](https://mlsanity.com/paper/1511.02222/) (2015)
- [Scalable Variational Gaussian Process Classification](https://mlsanity.com/paper/1411.2005/) (2014)
- [When Gaussian Process Meets Big Data: A Review of Scalable GPs](https://mlsanity.com/paper/1807.01065/) (2018)
- [Kernel Interpolation for Scalable Structured Gaussian Processes (KISS-GP)](https://mlsanity.com/paper/1503.01057/) (2015)
- [Distributed Gaussian Processes](https://mlsanity.com/paper/1502.02843/) (2015)
- [Manifold Gaussian Processes for Regression](https://mlsanity.com/paper/1402.5876/) (2014)
- [Hierarchical Nearest-Neighbor Gaussian Process Models for Large Geostatistical Datasets](https://mlsanity.com/paper/1406.7343/) (2014)
- [A general framework for Vecchia approximations of Gaussian processes](https://mlsanity.com/paper/1708.06302/) (2017)
- [A Case Study Competition Among Methods for Analyzing Large Spatial Data](https://mlsanity.com/paper/1710.05013/) (2017)
- [Sparse Cholesky factorization by Kullback-Leibler minimization](https://mlsanity.com/paper/2004.14455/) (2020)
- [Active Learning of Linear Embeddings for Gaussian Processes](https://mlsanity.com/paper/1310.6740/) (2013)
- [Permutation and Grouping Methods for Sharpening Gaussian Process Approximations](https://mlsanity.com/paper/1609.05372/) (2016)
- [Compression, inversion, and approximate PCA of dense kernel matrices at near-linear computational complexity](https://mlsanity.com/paper/1706.02205/) (2017)
- [Vecchia approximations of Gaussian-process predictions](https://mlsanity.com/paper/1805.03309/) (2018)
- [A multi-resolution approximation for massive spatial datasets](https://mlsanity.com/paper/1507.04789/) (2015)
- [Randomly Projected Additive Gaussian Processes for Regression](https://mlsanity.com/paper/1912.12834/) (2019)
- [Scaled Vecchia approximation for fast computer-model emulation](https://mlsanity.com/paper/2005.00386/) (2020)
- [Gaussian Process Learning via Fisher Scoring of Vecchia's Approximation](https://mlsanity.com/paper/1905.08374/) (2019)
- [Scalable Bayesian transport maps for high-dimensional non-Gaussian spatial fields](https://mlsanity.com/paper/2108.04211/) (2021)
- [Iteratively reweighted adaptive lasso for conditional heteroscedastic time series with applications to AR-ARCH type processes](https://mlsanity.com/paper/1502.06557/) (2015)
- [A class of multi-resolution approximations for large spatial datasets](https://mlsanity.com/paper/1710.08976/) (2017)

## Built on by (most cited first, 8 of 8)

- [Variational sparse inverse Cholesky approximation for latent Gaussian processes via double Kullback-Leibler minimization](https://mlsanity.com/paper/2301.13303/) (2023)
- [Vecchia Gaussian Process Ensembles on Internal Representations of Deep Neural Networks](https://mlsanity.com/paper/2305.17063/) (2023)
- [A model aggregation approach for high-dimensional large-scale optimization](https://mlsanity.com/paper/2205.07525/) (2022)
- [Combining additivity and active subspaces for high-dimensional Gaussian process modeling](https://mlsanity.com/paper/2402.03809/) (2024)
- [Adaptive Replication Strategies in Trust-Region-Based Bayesian Optimization of Stochastic Functions](https://mlsanity.com/paper/2504.20527/) (2025)
- [Bayesian Optimization for Dynamic Pricing and Learning](https://mlsanity.com/paper/2510.12447/) (2025)
- [Scalable Derivative Gaussian Processes via Exact Gradient Reduction](https://mlsanity.com/paper/2606.02909/) (2026)
- [Sparsity by Default: The Theory and Practice of ARD in Gaussian Process Regression for Variable Selection](https://mlsanity.com/paper/2609.33550/) (2026)

## Abstract (arXiv)

Gaussian process (GP) regression is a flexible, nonparametric approach to regression that naturally quantifies uncertainty. In many applications, the number of responses and covariates are both large, and a goal is to select covariates that are related to the response. For this setting, we propose a novel, scalable algorithm, coined VGPR, which optimizes a penalized GP log-likelihood based on the Vecchia GP approximation, an ordered conditional approximation from spatial statistics that implies a sparse Cholesky factor of the precision matrix. We traverse the regularization path from strong to weak penalization, sequentially adding candidate covariates based on the gradient of the log-likelihood and deselecting irrelevant covariates via a new quadratic constrained coordinate descent algorithm. We propose Vecchia-based mini-batch subsampling, which provides unbiased gradient estimators. The resulting procedure is scalable to millions of responses and thousands of covariates. Theoretical analysis and numerical studies demonstrate the improved scalability and accuracy relative to existing methods.