# A Unifying Perspective on Non-Stationary Kernels for Deeper Gaussian Processes

arXiv: https://arxiv.org/abs/2309.10068 · HTML: https://mlsanity.com/paper/2309.10068/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 92 references that are in the map.
- First posted on arXiv 18 Sep 2023.

## Builds on (most cited first, 26 of 26)

- [GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration](https://mlsanity.com/paper/1809.11165/) (2018)
- [Deep Kernel Learning](https://mlsanity.com/paper/1511.02222/) (2015)
- [Deep learning: a statistical viewpoint](https://mlsanity.com/paper/2103.09177/) (2021)
- [Scalable Variational Gaussian Process Classification](https://mlsanity.com/paper/1411.2005/) (2014)
- [Doubly Stochastic Variational Inference for Deep Gaussian Processes](https://mlsanity.com/paper/1705.08933/) (2017)
- [Inconsistency of Bayesian Inference for Misspecified Linear Models, and a Proposal for Repairing It](https://mlsanity.com/paper/1412.3730/) (2014)
- [The Promises and Pitfalls of Deep Kernel Learning](https://mlsanity.com/paper/2102.12108/) (2021)
- [Local Gaussian process approximation for large computer experiments](https://mlsanity.com/paper/1303.0383/) (2013)
- [Random Feature Expansions for Deep Gaussian Processes](https://mlsanity.com/paper/1610.04386/) (2016)
- [Inference in Deep Gaussian Processes using Stochastic Gradient Hamiltonian Monte Carlo](https://mlsanity.com/paper/1806.05490/) (2018)
- [Sampling from Gaussian Process Posteriors using Stochastic Gradient Descent](https://mlsanity.com/paper/2306.11589/) (2023)
- [Deep convolutional Gaussian processes](https://mlsanity.com/paper/1810.03052/) (2018)
- [Variational Auto-encoded Deep Gaussian Processes](https://mlsanity.com/paper/1511.06455/) (2015)
- [Deep Compositional Spatial Models](https://mlsanity.com/paper/1906.02840/) (2019)
- [Deep Gaussian Processes with Importance-Weighted Variational Inference](https://mlsanity.com/paper/1905.05435/) (2019)
- [GPflux: A Library for Deep Gaussian Processes](https://mlsanity.com/paper/2104.05674/) (2021)
- [Non-Stationary Spatial Modeling](https://mlsanity.com/paper/2212.08043/) (2022)
- [Deep Exponential Families](https://mlsanity.com/paper/1411.2581/) (2014)
- [Exact Gaussian Processes for Massive Datasets via Non-Stationary Sparsity-Discovering Kernels](https://mlsanity.com/paper/2205.09070/) (2022)
- [Scalable Gaussian Process Computations Using Hierarchical Matrices](https://mlsanity.com/paper/1808.03215/) (2018)
- [Nonstationary Gaussian Process Surrogates](https://mlsanity.com/paper/2305.19242/) (2023)
- [Advanced Stationary and Non-Stationary Kernel Designs for Domain-Aware Gaussian Processes](https://mlsanity.com/paper/2102.03432/) (2021)
- [Sharded Bayesian Additive Regression Trees](https://mlsanity.com/paper/2306.00361/) (2023)
- [Regression-based covariance functions for nonstationary spatial modeling](https://mlsanity.com/paper/1410.1494/) (2014)
- [Vecchia Gaussian Process Ensembles on Internal Representations of Deep Neural Networks](https://mlsanity.com/paper/2305.17063/) (2023)
- [Nonparametric Multi-shape Modeling with Uncertainty Quantification](https://mlsanity.com/paper/2206.09127/) (2022)

## Built on by (most cited first, 8 of 8)

- [Kernel-based estimators for functional causal effects](https://mlsanity.com/paper/2503.05024/) (2025)
- [Compactly-supported nonstationary kernels for computing exact Gaussian processes on big data](https://mlsanity.com/paper/2411.05869/) (2024)
- [GGMPs: Generalized Gaussian Mixture Processes](https://mlsanity.com/paper/2603.10442/) (2026)
- [A New Reliable & Parsimonious Learning Strategy Comprising Two Layers of Gaussian Processes, to Address Inhomogeneous Empirical Correlation Structures](https://mlsanity.com/paper/2404.12478/) (2024)
- [SEEK: Self-adaptive Explainable Kernel For Nonstationary Gaussian Processes](https://mlsanity.com/paper/2503.14785/) (2025)
- [Parametrising the Inhomogeneity Inducing Capacity of a Training Set, and its Impact on Supervised Learning](https://mlsanity.com/paper/2510.18332/) (2025)
- [Bayesian Optimization of Genetic Algorithm Hyperparameters in a Multi-Fidelity Framework for Efficient Lattice Material Design](https://mlsanity.com/paper/2607.07289/) (2026)
- [Covariance-Boosted Gaussian Processes for Spatiotemporal Irregularities](https://mlsanity.com/paper/2607.23018/) (2026)

## Abstract (arXiv)

The Gaussian process (GP) is a popular statistical technique for stochastic function approximation and uncertainty quantification from data. GPs have been adopted into the realm of machine learning in the last two decades because of their superior prediction abilities, especially in data-sparse scenarios, and their inherent ability to provide robust uncertainty estimates. Even so, their performance highly depends on intricate customizations of the core methodology, which often leads to dissatisfaction among practitioners when standard setups and off-the-shelf software tools are being deployed. Arguably the most important building block of a GP is the kernel function which assumes the role of a covariance operator. Stationary kernels of the Mat'ern class are used in the vast majority of applied studies; poor prediction performance and unrealistic uncertainty quantification are often the consequences. Non-stationary kernels show improved performance but are rarely used due to their more complicated functional form and the associated effort and expertise needed to define and tune them optimally. In this perspective, we want to help ML practitioners make sense of some of the most common forms of non-stationarity for Gaussian processes. We show a variety of kernels in action using representative datasets, carefully study their properties, and compare their performances. Based on our findings, we propose a new kernel that combines some of the identified advantages of existing kernels.