# Thoughts on Massively Scalable Gaussian Processes

arXiv: https://arxiv.org/abs/1511.01870 · HTML: https://mlsanity.com/paper/1511.01870/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Nov 2015.

## Built on by (most cited first, 26 of 26)

- [The Promises and Pitfalls of Deep Kernel Learning](https://mlsanity.com/paper/2102.12108/) (2021)
- [Posterior and Computational Uncertainty in Gaussian Processes](https://mlsanity.com/paper/2205.15449/) (2022)
- [Preconditioning for Scalable Gaussian Process Hyperparameter Optimization](https://mlsanity.com/paper/2107.00243/) (2021)
- [Local approximate Gaussian process regression for data-driven constitutive laws: Development and comparison with neural networks](https://mlsanity.com/paper/2105.04554/) (2021)
- [Diverse Video Generation using a Gaussian Process Trigger](https://mlsanity.com/paper/2107.04619/) (2021)
- [Solving High Frequency and Multi-Scale PDEs with Gaussian Processes](https://mlsanity.com/paper/2311.04465/) (2023)
- [SKIing on Simplices: Kernel Interpolation on the Permutohedral Lattice for Scalable Gaussian Processes](https://mlsanity.com/paper/2106.06695/) (2021)
- [Scalable Variational Gaussian Processes via Harmonic Kernel Decomposition](https://mlsanity.com/paper/2106.05992/) (2021)
- [Multitask methods for predicting molecular properties from heterogeneous data](https://mlsanity.com/paper/2401.17898/) (2024)
- [DKL-KAN: Scalable Deep Kernel Learning using Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2407.21176/) (2024)
- [Toward Efficient Kernel-Based Solvers for Nonlinear PDEs](https://mlsanity.com/paper/2410.11165/) (2024)
- [Hierarchical Inducing Point Gaussian Process for Inter-domain Observations](https://mlsanity.com/paper/2103.00393/) (2021)
- [Semi-Parametric Inducing Point Networks and Neural Processes](https://mlsanity.com/paper/2205.11718/) (2022)
- [Faster Kernel Interpolation for Gaussian Processes](https://mlsanity.com/paper/2101.11751/) (2021)
- [Federated Bayesian Neural Regression: A Scalable Global Federated Gaussian Process](https://mlsanity.com/paper/2206.06357/) (2022)
- [Sparse Kernel Gaussian Processes through Iterative Charted Refinement (ICR)](https://mlsanity.com/paper/2206.10634/) (2022)
- [Recommendations for Baselines and Benchmarking Approximate Gaussian Processes](https://mlsanity.com/paper/2402.09849/) (2024)
- [Scaling up Probabilistic PDE Simulators with Structured Volumetric Information](https://mlsanity.com/paper/2406.05020/) (2024)
- [HiGP: A high-performance Python package for Gaussian Process](https://mlsanity.com/paper/2503.02259/) (2025)
- [Distilled Thompson Sampling: Practical and Efficient Thompson Sampling via Imitation Learning](https://mlsanity.com/paper/2011.14266/) (2020)
- [Bayesian Inference in High-Dimensional Time-Serieswith the Orthogonal Stochastic Linear Mixing Model](https://mlsanity.com/paper/2106.13379/) (2021)
- [Generalized Least Squares Kernelized Tensor Factorization](https://mlsanity.com/paper/2412.07041/) (2024)
- [Tensor Gaussian Processes: Efficient Solvers for Nonlinear PDEs](https://mlsanity.com/paper/2510.13772/) (2025)
- [gp2Scale: A Class of Compactly Supported Non-Stationary Kernels and Distributed Computing for Exact Gaussian Processes on 10 Million Data Points](https://mlsanity.com/paper/2512.06143/) (2025)
- [Forward variable selection enables fast and accurate dynamic system identification with Karhunen-Lo`eve decomposed Gaussian processes](https://mlsanity.com/paper/2205.13676/) (2022)
- [Large-scale magnetic field maps using structured kernel interpolation for Gaussian process regression](https://mlsanity.com/paper/2310.16574/) (2023)

## Abstract (arXiv)

We introduce a framework and early results for massively scalable Gaussian processes (MSGP), significantly extending the KISS-GP approach of Wilson and Nickisch (2015). The MSGP framework enables the use of Gaussian processes (GPs) on billions of datapoints, without requiring distributed inference, or severe assumptions. In particular, MSGP reduces the standard O(n3) complexity of GP learning and inference to O(n), and the standard O(n2) complexity per test point prediction to O(1). MSGP involves 1) decomposing covariance matrices as Kronecker products of Toeplitz matrices approximated by circulant matrices. This multi-level circulant approximation allows one to unify the orthogonal computational benefits of fast Kronecker and Toeplitz approaches, and is significantly faster than either approach in isolation; 2) local kernel interpolation and inducing points to allow for arbitrarily located data inputs, and O(1) test time predictions; 3) exploiting block-Toeplitz Toeplitz-block structure (BTTB), which enables fast inference and learning when multidimensional Kronecker structure is not present; and 4) projections of the input space to flexibly model correlated inputs and high dimensional data. The ability to handle many (m ≈ n) inducing points allows for near-exact accuracy and large scale kernel learning.