# An analytic comparison of regularization methods for Gaussian Processes

arXiv: https://arxiv.org/abs/1602.00853 · HTML: https://mlsanity.com/paper/1602.00853/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Feb 2016.

## Built on by (most cited first, 7 of 7)

- [Quantum Gaussian Process Regression for Bayesian Optimization](https://mlsanity.com/paper/2304.12923/) (2023)
- [Principled Bayesian Optimisation in Collaboration with Human Experts](https://mlsanity.com/paper/2410.10452/) (2024)
- [Bayesian optimization with improved scalability and derivative information for efficient design of nanophotonic structures](https://mlsanity.com/paper/2101.02972/) (2021)
- [Exploring the Links between the Fundamental Lemma and Kernel Regression](https://mlsanity.com/paper/2403.05368/) (2024)
- [Gearing Gaussian process modeling and sequential design towards stochastic simulators](https://mlsanity.com/paper/2412.07306/) (2024)
- [Multidimensional Distributional Neural Network Output Demonstrated in Super-Resolution of Surface Wind Speed](https://mlsanity.com/paper/2508.16686/) (2025)
- [A Quadratic Order Reduction -- Gaussian Process Ordinary Differential Equation framework for the inference of Large Continuous Dynamical Systems](https://mlsanity.com/paper/2606.13063/) (2026)

## Abstract (arXiv)

Gaussian Processes (GPs) are a popular approach to predict the output of a parameterized experiment. They have many applications in the field of Computer Experiments, in particular to perform sensitivity analysis, adaptive design of experiments and global optimization. Nearly all of the applications of GPs require the inversion of a covariance matrix that, in practice, is often ill-conditioned. Regularization methodologies are then employed with consequences on the GPs that need to be better understood.The two principal methods to deal with ill-conditioned covariance matrices are i) pseudoinverse and ii) adding a positive constant to the diagonal (the so-called nugget regularization).The first part of this paper provides an algebraic comparison of PI and nugget regularizations. Redundant points, responsible for covariance matrix singularity, are defined. It is proven that pseudoinverse regularization, contrarily to nugget regularization, averages the output values and makes the variance zero at redundant points. However, pseudoinverse and nugget regularizations become equivalent as the nugget value vanishes. A measure for data-model discrepancy is proposed which serves for choosing a regularization technique.In the second part of the paper, a distribution-wise GP is introduced that interpolates Gaussian distributions instead of data points. Distribution-wise GP can be seen as an improved regularization method for GPs.