# Physics Inspired Approaches To Understanding Gaussian Processes

arXiv: https://arxiv.org/abs/2305.10748 · HTML: https://mlsanity.com/paper/2305.10748/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 52 references that are in the map.
- First posted on arXiv 18 May 2023.

## Builds on (most cited first, 24 of 24)

- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [Deep Kernel Learning](https://mlsanity.com/paper/1511.02222/) (2015)
- [Deep Ensembles: A Loss Landscape Perspective](https://mlsanity.com/paper/1912.02757/) (2019)
- [Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges](https://mlsanity.com/paper/2103.11251/) (2021)
- [The Loss Surfaces of Multilayer Networks](https://mlsanity.com/paper/1412.0233/) (2014)
- [When Gaussian Process Meets Big Data: A Review of Scalable GPs](https://mlsanity.com/paper/1807.01065/) (2018)
- [Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes](https://mlsanity.com/paper/1706.10239/) (2017)
- [Mat'ern Gaussian processes on Riemannian manifolds](https://mlsanity.com/paper/2006.10160/) (2020)
- [The Global Landscape of Neural Networks: An Overview](https://mlsanity.com/paper/2007.01429/) (2020)
- [Bandit optimisation of functions in the Mat'ern kernel RKHS](https://mlsanity.com/paper/2001.10396/) (2020)
- [Unveiling the structure of wide flat minima in neural networks](https://mlsanity.com/paper/2107.01163/) (2021)
- [Energy-entropy competition and the effectiveness of stochastic gradient descent in machine learning](https://mlsanity.com/paper/1803.01927/) (2018)
- [Function-Space Distributions over Kernels](https://mlsanity.com/paper/1910.13565/) (2019)
- [Exploration-Exploitation in Multi-Agent Learning: Catastrophe Theory Meets Game Theory](https://mlsanity.com/paper/2012.03083/) (2020)
- [Incremental Ensemble Gaussian Processes](https://mlsanity.com/paper/2110.06777/) (2021)
- [Visualising Basins of Attraction for the Cross-Entropy and the Squared Error Neural Network Loss Functions](https://mlsanity.com/paper/1901.02302/) (2019)
- [Tilting the playing field: Dynamical loss functions for machine learning](https://mlsanity.com/paper/2102.03793/) (2021)
- [Gaussian Process Regression with Local Explanation](https://mlsanity.com/paper/2007.01669/) (2020)
- [Measuring the robustness of Gaussian processes to kernel choice](https://mlsanity.com/paper/2106.06510/) (2021)
- [Evaluating Sensitivity to the Stick-Breaking Prior in Bayesian Nonparametrics](https://mlsanity.com/paper/2107.03584/) (2021)
- [Attacking Adversarial Defences by Smoothing the Loss Landscape](https://mlsanity.com/paper/2208.00862/) (2022)
- [Fitting Mat'ern Smoothness Parameters Using Automatic Differentiation](https://mlsanity.com/paper/2201.00090/) (2022)
- [Deep Ensemble as a Gaussian Process Approximate Posterior](https://mlsanity.com/paper/2205.00163/) (2022)

## Abstract (arXiv)

Prior beliefs about the latent function to shape inductive biases can be incorporated into a Gaussian Process (GP) via the kernel. However, beyond kernel choices, the decision-making process of GP models remains poorly understood. In this work, we contribute an analysis of the loss landscape for GP models using methods from physics. We demonstrate nu-continuity for Matern kernels and outline aspects of catastrophe theory at critical points in the loss landscape. By directly including nu in the hyperparameter optimisation for Matern kernels, we find that typical values of nu are far from optimal in terms of performance, yet prevail in the literature due to the increased computational speed. We also provide an a priori method for evaluating the effect of GP ensembles and discuss various voting approaches based on physical properties of the loss landscape. The utility of these approaches is demonstrated for various synthetic and real datasets. Our findings provide an enhanced understanding of the decision-making process behind GPs and offer practical guidance for improving their performance and interpretability in a range of applications.