# Maximum Likelihood Estimation in Gaussian Process Regression is Ill-Posed

arXiv: https://arxiv.org/abs/2203.09179 · HTML: https://mlsanity.com/paper/2203.09179/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 86 references that are in the map.
- First posted on arXiv 17 Mar 2022.

## Builds on (most cited first, 11 of 11)

- [Deep Kernel Learning](https://mlsanity.com/paper/1511.02222/) (2015)
- [Gaussian Processes and Kernel Methods: A Review on Connections and Equivalences](https://mlsanity.com/paper/1807.02582/) (2018)
- [GPflow: A Gaussian process library using TensorFlow](https://mlsanity.com/paper/1610.08733/) (2016)
- [Bayesian Probabilistic Numerical Methods](https://mlsanity.com/paper/1702.03673/) (2017)
- [On the marginal likelihood and cross-validation](https://mlsanity.com/paper/1905.08737/) (2019)
- [Convergence Guarantees for Gaussian Process Means With Misspecified Likelihoods and Smoothness](https://mlsanity.com/paper/2001.10818/) (2020)
- [Overview of Gaussian process based multi-fidelity techniques with variable relationship between fidelities](https://mlsanity.com/paper/2006.16728/) (2020)
- [Monte Carlo with Determinantal Point Processes](https://mlsanity.com/paper/1605.00361/) (2016)
- [On the well-posedness of Bayesian inverse problems](https://mlsanity.com/paper/1902.10257/) (2019)
- [Consistency of Empirical Bayes And Kernel Flow For Hierarchical Parameter Estimation](https://mlsanity.com/paper/2005.11375/) (2020)
- [Maximum likelihood estimation and uncertainty quantification for Gaussian process approximation of deterministic functions](https://mlsanity.com/paper/2001.10965/) (2020)

## Built on by (most cited first, 16 of 16)

- [Vanilla Bayesian Optimization Performs Great in High Dimensions](https://mlsanity.com/paper/2402.02229/) (2024)
- [Understanding High-Dimensional Bayesian Optimization](https://mlsanity.com/paper/2502.09198/) (2025)
- [On Safety in Safe Bayesian Optimization](https://mlsanity.com/paper/2403.12948/) (2024)
- [Transfer Learning for Bayesian Optimization on Heterogeneous Search Spaces](https://mlsanity.com/paper/2309.16597/) (2023)
- [HyperBO+: Pre-training a universal prior for Bayesian optimization with hierarchical Gaussian processes](https://mlsanity.com/paper/2212.10538/) (2022)
- [Probabilistic Neural Networks (PNNs) for Modeling Aleatoric Uncertainty in Scientific Machine Learning](https://mlsanity.com/paper/2402.13945/) (2024)
- [Wasserstein Barycenter Gaussian Process based Bayesian Optimization](https://mlsanity.com/paper/2505.12471/) (2025)
- [Physics-based Reduced Order Modeling for Uncertainty Quantification of Guided Wave Propagation using Bayesian Optimization](https://mlsanity.com/paper/2307.09661/) (2023)
- [Multidimensional Distributional Neural Network Output Demonstrated in Super-Resolution of Surface Wind Speed](https://mlsanity.com/paper/2508.16686/) (2025)
- [Bayesian Quadrature](https://mlsanity.com/paper/2602.16218/) (2026)
- [Active Discrimination Learning for Gaussian Process Models](https://mlsanity.com/paper/2211.11624/) (2022)
- [Flag Aggregator: Scalable Distributed Training under Failures and Augmented Losses using Convex Optimization](https://mlsanity.com/paper/2302.05865/) (2023)
- [On the Laplace Approximation as Model Selection Criterion for Gaussian Processes](https://mlsanity.com/paper/2403.09215/) (2024)
- [Weighted Leave-One-Out Cross Validation](https://mlsanity.com/paper/2505.19737/) (2025)
- [Scalable multitask Gaussian processes for complex mechanical systems with functional covariates](https://mlsanity.com/paper/2602.20640/) (2026)
- [In-Context Guidance: Learning Inter-Task Synergies via Numerical Foundational Models for Few-Shot Multitask Optimization](https://mlsanity.com/paper/2609.25836/) (2026)

## Abstract (arXiv)

Gaussian process regression underpins countless academic and industrial applications of machine learning and statistics, with maximum likelihood estimation routinely used to select appropriate parameters for the covariance kernel. However, it remains an open problem to establish the circumstances in which maximum likelihood estimation is well-posed, that is, when the predictions of the regression model are insensitive to small perturbations of the data. This article identifies scenarios where the maximum likelihood estimator fails to be well-posed, in that the predictive distributions are not Lipschitz in the data with respect to the Hellinger distance. These failure cases occur in the noiseless data setting, for any Gaussian process with a stationary covariance function whose lengthscale parameter is estimated using maximum likelihood. Although the failure of maximum likelihood estimation is part of Gaussian process folklore, these rigorous theoretical results appear to be the first of their kind. The implication of these negative results is that well-posedness may need to be assessed post-hoc, on a case-by-case basis, when maximum likelihood estimation is used to train a Gaussian process model.