# Scalable Cross Validation Losses for Gaussian Process Models

arXiv: https://arxiv.org/abs/2105.11535 · HTML: https://mlsanity.com/paper/2105.11535/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 42 references that are in the map.
- First posted on arXiv 24 May 2021.

## Builds on (most cited first, 22 of 22)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [Gaussian Processes for Big Data](https://mlsanity.com/paper/1309.6835/) (2013)
- [Pyro: Deep Universal Probabilistic Programming](https://mlsanity.com/paper/1810.09538/) (2018)
- [Scalable Variational Gaussian Process Classification](https://mlsanity.com/paper/1411.2005/) (2014)
- [When Gaussian Process Meets Big Data: A Review of Scalable GPs](https://mlsanity.com/paper/1807.01065/) (2018)
- [Exact Gaussian Processes on a Million Data Points](https://mlsanity.com/paper/1903.08114/) (2019)
- [Hierarchical Nearest-Neighbor Gaussian Process Models for Large Geostatistical Datasets](https://mlsanity.com/paper/1406.7343/) (2014)
- [Understanding Probabilistic Sparse Gaussian Process Approximations](https://mlsanity.com/paper/1606.04820/) (2016)
- [A general framework for Vecchia approximations of Gaussian processes](https://mlsanity.com/paper/1708.06302/) (2017)
- [Local Gaussian process approximation for large computer experiments](https://mlsanity.com/paper/1303.0383/) (2013)
- [On the marginal likelihood and cross-validation](https://mlsanity.com/paper/1905.08737/) (2019)
- [Dependent Multinomial Models Made Easy: Stick Breaking with the P'olya-Gamma Augmentation](https://mlsanity.com/paper/1506.05843/) (2015)
- [Sparse within Sparse Gaussian Processes using Neighbor Information](https://mlsanity.com/paper/2011.05041/) (2020)
- [Efficient Gaussian Process Classification Using Polya-Gamma Data Augmentation](https://mlsanity.com/paper/1802.06383/) (2018)
- [Deep Sigma Point Processes](https://mlsanity.com/paper/2002.09112/) (2020)
- [Multi-Class Gaussian Process Classification Made Conjugate: Efficient Inference via Data Augmentation](https://mlsanity.com/paper/1905.09670/) (2019)
- [Direct loss minimization algorithms for sparse Gaussian processes](https://mlsanity.com/paper/2004.03083/) (2020)
- [Bayesian leave-one-out cross-validation approximations for Gaussian latent variable models](https://mlsanity.com/paper/1412.7461/) (2014)
- [Towards new cross-validation-based estimators for Gaussian process regression: efficient adjoint computation of gradients](https://mlsanity.com/paper/2002.11543/) (2020)
- [Differentially Private Gaussian Processes](https://mlsanity.com/paper/1606.00720/) (2016)
- [Sparse Gaussian Processes via Parametric Families of Compactly-supported Kernels](https://mlsanity.com/paper/2006.03673/) (2020)

## Built on by (most cited first, 6 of 6)

- [Computation-Aware Gaussian Processes: Model Selection And Linear-Time Inference](https://mlsanity.com/paper/2411.01036/) (2024)
- [Variational Nearest Neighbor Gaussian Process](https://mlsanity.com/paper/2202.01694/) (2022)
- [Fast and Scalable Spike and Slab Variable Selection in High-Dimensional Gaussian Processes](https://mlsanity.com/paper/2111.04558/) (2021)
- [Gaussian Process Inference Using Mini-batch Stochastic Gradient Descent: Convergence Guarantees and Empirical Benefits](https://mlsanity.com/paper/2111.10461/) (2021)
- [Interventional Processes for Causal Uncertainty Quantification](https://mlsanity.com/paper/2410.14483/) (2024)
- [On the Calibration and Uncertainty with P'olya-Gamma Augmentation for Dialog Retrieval Models](https://mlsanity.com/paper/2303.08606/) (2023)

## Abstract (arXiv)

We introduce a simple and scalable method for training Gaussian process (GP) models that exploits cross-validation and nearest neighbor truncation. To accommodate binary and multi-class classification we leverage P`olya-Gamma auxiliary variables and variational inference. In an extensive empirical comparison with a number of alternative methods for scalable GP regression and classification, we find that our method offers fast training and excellent predictive performance. We argue that the good predictive performance can be traced to the non-parametric nature of the resulting predictive distributions as well as to the cross-validation loss, which provides robustness against model mis-specification.