# When are Iterative Gaussian Processes Reliably Accurate?

arXiv: https://arxiv.org/abs/2112.15246 · HTML: https://mlsanity.com/paper/2112.15246/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 25 references that are in the map.
- First posted on arXiv 31 Dec 2021.

## Builds on (most cited first, 12 of 12)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration](https://mlsanity.com/paper/1809.11165/) (2018)
- [Gaussian Processes for Big Data](https://mlsanity.com/paper/1309.6835/) (2013)
- [Exact Gaussian Processes on a Million Data Points](https://mlsanity.com/paper/1903.08114/) (2019)
- [Hilbert Space Methods for Reduced-Rank Gaussian Process Regression](https://mlsanity.com/paper/1401.5508/) (2014)
- [Constant-Time Predictive Distributions for Gaussian Processes](https://mlsanity.com/paper/1803.06058/) (2018)
- [Preconditioning Kernel Matrices](https://mlsanity.com/paper/1602.06693/) (2016)
- [Fast Matrix Square Roots with Applications to Gaussian Processes and Bayesian Optimization](https://mlsanity.com/paper/2006.11267/) (2020)
- [Tighter Bounds on the Log Marginal Likelihood of Gaussian Process Regression Using Conjugate Gradients](https://mlsanity.com/paper/2102.08314/) (2021)
- [Bias-Free Scalable Gaussian Processes via Randomized Truncations](https://mlsanity.com/paper/2102.06695/) (2021)
- [SKIing on Simplices: Kernel Interpolation on the Permutohedral Lattice for Scalable Gaussian Processes](https://mlsanity.com/paper/2106.06695/) (2021)
- [Scalable Variational Gaussian Processes via Harmonic Kernel Decomposition](https://mlsanity.com/paper/2106.05992/) (2021)

## Built on by (most cited first, 12 of 12)

- [Parallel MCMC Without Embarrassing Failures](https://mlsanity.com/paper/2202.11154/) (2022)
- [Low-Precision Arithmetic for Fast Gaussian Processes](https://mlsanity.com/paper/2207.06856/) (2022)
- [Iterative Methods for Vecchia-Laplace Approximations for Latent Gaussian Process Models](https://mlsanity.com/paper/2310.12000/) (2023)
- [Improving Linear System Solvers for Hyperparameter Optimisation in Iterative Gaussian Processes](https://mlsanity.com/paper/2405.18457/) (2024)
- [Graph Random Features for Scalable Gaussian Processes](https://mlsanity.com/paper/2509.03691/) (2025)
- [Iterative Methods for Full-Scale Gaussian Process Approximations for Large Spatial Data](https://mlsanity.com/paper/2405.14492/) (2024)
- [Recommendations for Baselines and Benchmarking Approximate Gaussian Processes](https://mlsanity.com/paper/2402.09849/) (2024)
- [High-Dimensional Gaussian Process Regression with Soft Kernel Interpolation](https://mlsanity.com/paper/2410.21419/) (2024)
- [Kernel Multigrid: Accelerate Back-fitting via Sparse Gaussian Process Regression](https://mlsanity.com/paper/2403.13300/) (2024)
- [Review of Recent Advances in Gaussian Process Regression Methods](https://mlsanity.com/paper/2409.08112/) (2024)
- [Fast Gaussian process inference by exact Mat'ern kernel decomposition](https://mlsanity.com/paper/2508.01864/) (2025)
- [Warm-Starting Iterative Gaussian Processes for Faster Sequential Inference](https://mlsanity.com/paper/2511.16340/) (2025)

## Abstract (arXiv)

While recent work on conjugate gradient methods and Lanczos decompositions have achieved scalable Gaussian process inference with highly accurate point predictions, in several implementations these iterative methods appear to struggle with numerical instabilities in learning kernel hyperparameters, and poor test likelihoods. By investigating CG tolerance, preconditioner rank, and Lanczos decomposition rank, we provide a particularly simple prescription to correct these issues: we recommend that one should use a small CG tolerance (ε ≤ 0.01) and a large root decomposition size (r ≥ 5000). Moreover, we show that L-BFGS-B is a compelling optimizer for Iterative GPs, achieving convergence with fewer gradient updates.