mlsanityData to 14 Oct 2026

When are Iterative Gaussian Processes Reliably Accurate?

arXiv 2112.15246 ↗First posted 31 Dec 2021cs.LG stat.MLcited by 12

Not on the map. Why?

Abstract (arXiv)

While recent work on conjugate gradient methods and Lanczos decompositions have achieved scalable Gaussian process inference with highly accurate point predictions, in several implementations these iterative methods appear to struggle with numerical instabilities in learning kernel hyperparameters, and poor test likelihoods. By investigating CG tolerance, preconditioner rank, and Lanczos decomposition rank, we provide a particularly simple prescription to correct these issues: we recommend that one should use a small CG tolerance (ε ≤ 0.01) and a large root decomposition size (r ≥ 5000). Moreover, we show that L-BFGS-B is a compelling optimizer for Iterative GPs, achieving convergence with fewer gradient updates.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 12 of its 25 references that are in the map. First posted on arXiv 31 Dec 2021.

Builds on

12 of 25 references.

References · 12

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2018GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Accelerationcited by 609
  3. 2013Gaussian Processes for Big Datacited by 383
  4. 2019Exact Gaussian Processes on a Million Data Pointscited by 119
  5. 2014Hilbert Space Methods for Reduced-Rank Gaussian Process Regressioncited by 88
  6. 2018Constant-Time Predictive Distributions for Gaussian Processescited by 48
  7. 2016Preconditioning Kernel Matricescited by 33
  8. 2020Fast Matrix Square Roots with Applications to Gaussian Processes and Bayesian Optimizationcited by 26
  9. 2021Tighter Bounds on the Log Marginal Likelihood of Gaussian Process Regression Using Conjugate Gradientscited by 20
  10. 2021Bias-Free Scalable Gaussian Processes via Randomized Truncationscited by 16
  11. 2021SKIing on Simplices: Kernel Interpolation on the Permutohedral Lattice for Scalable Gaussian Processescited by 9
  12. 2021Scalable Variational Gaussian Processes via Harmonic Kernel Decompositioncited by 8

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 1; 2024: 6; 2025: 3.

12 papers.

  1. 2022Parallel MCMC Without Embarrassing Failurescited by 11
  2. 2022Low-Precision Arithmetic for Fast Gaussian Processescited by 10
  3. 2023Iterative Methods for Vecchia-Laplace Approximations for Latent Gaussian Process Modelscited by 6
  4. 2024Improving Linear System Solvers for Hyperparameter Optimisation in Iterative Gaussian Processescited by 5
  5. 2025Graph Random Features for Scalable Gaussian Processescited by 5
  6. 2024Iterative Methods for Full-Scale Gaussian Process Approximations for Large Spatial Datacited by 4
  7. 2024Recommendations for Baselines and Benchmarking Approximate Gaussian Processescited by 2
  8. 2024High-Dimensional Gaussian Process Regression with Soft Kernel Interpolationcited by 1
  9. 2024Kernel Multigrid: Accelerate Back-fitting via Sparse Gaussian Process Regressioncited by 0
  10. 2024Review of Recent Advances in Gaussian Process Regression Methodscited by 0
  11. 2025Fast Gaussian process inference by exact Mat'ern kernel decompositioncited by 0
  12. 2025Warm-Starting Iterative Gaussian Processes for Faster Sequential Inferencecited by 0