# On the Laplace Approximation as Model Selection Criterion for Gaussian Processes

arXiv: https://arxiv.org/abs/2403.09215 · HTML: https://mlsanity.com/paper/2403.09215/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 45 references that are in the map.
- First posted on arXiv 14 Mar 2024.

## Builds on (most cited first, 18 of 18)

- [GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration](https://mlsanity.com/paper/1809.11165/) (2018)
- [Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting](https://mlsanity.com/paper/1805.07810/) (2018)
- [Gaussian Process Kernels for Pattern Discovery and Extrapolation](https://mlsanity.com/paper/1302.4245/) (2013)
- [Structure Discovery in Nonparametric Regression through Compositional Kernel Search](https://mlsanity.com/paper/1302.4922/) (2013)
- [Mat'ern Gaussian processes on Riemannian manifolds](https://mlsanity.com/paper/2006.10160/) (2020)
- [Linearly constrained Gaussian processes](https://mlsanity.com/paper/1703.00787/) (2017)
- [Automatic Construction and Natural-Language Description of Nonparametric Regression Models](https://mlsanity.com/paper/1402.4304/) (2014)
- [Learnable Uncertainty under Laplace Approximations](https://mlsanity.com/paper/2010.02720/) (2020)
- [Equivariant Learning of Stochastic Fields: Gaussian Processes and Steerable Conditional Neural Processes](https://mlsanity.com/paper/2011.12916/) (2020)
- [Scaling up the Automatic Statistician: Scalable Structure Discovery using Gaussian Processes](https://mlsanity.com/paper/1706.02524/) (2017)
- [Gaussian Process Priors for Systems of Linear Partial Differential Equations with Constant Coefficients](https://mlsanity.com/paper/2212.14319/) (2022)
- [Maximum Likelihood Estimation in Gaussian Process Regression is Ill-Posed](https://mlsanity.com/paper/2203.09179/) (2022)
- [Dynamic nested sampling: an improved algorithm for parameter estimation and evidence calculation](https://mlsanity.com/paper/1704.03459/) (2017)
- [Implicit Kernel Learning](https://mlsanity.com/paper/1902.10214/) (2019)
- [Marginalised Gaussian Processes with Nested Sampling](https://mlsanity.com/paper/2010.16344/) (2020)
- [NCVX: A General-Purpose Optimization Solver for Constrained Machine and Deep Learning](https://mlsanity.com/paper/2210.00973/) (2022)
- [Vecchia-Laplace approximations of generalized Gaussian processes for big non-Gaussian spatial data](https://mlsanity.com/paper/1906.07828/) (2019)
- [Improving Hyperparameter Learning under Approximate Inference in Gaussian Process Models](https://mlsanity.com/paper/2306.04201/) (2023)

## Abstract (arXiv)

Model selection aims to find the best model in terms of accuracy, interpretability or simplicity, preferably all at once. In this work, we focus on evaluating model performance of Gaussian process models, i.e. finding a metric that provides the best trade-off between all those criteria. While previous work considers metrics like the likelihood, AIC or dynamic nested sampling, they either lack performance or have significant runtime issues, which severely limits applicability. We address these challenges by introducing multiple metrics based on the Laplace approximation, where we overcome a severe inconsistency occuring during naive application of the Laplace approximation. Experiments show that our metrics are comparable in quality to the gold standard dynamic nested sampling without compromising for computational speed. Our model selection criteria allow significantly faster and high quality model selection of Gaussian process models.