mlsanityData to 14 Oct 2026

Sampling from Gaussian Process Posteriors using Stochastic Gradient Descent

arXiv 2306.11589 ↗First posted 20 Jun 2023cs.LG stat.MLcited by 27

Not on the map. Why?

Abstract (arXiv)

Gaussian processes are a powerful framework for quantifying uncertainty and for sequential decision-making but are limited by the requirement of solving linear systems. In general, this has a cubic cost in dataset size and is sensitive to conditioning. We explore stochastic gradient algorithms as a computationally efficient method of approximately solving these linear systems: we develop low-variance optimization objectives for sampling from the posterior and extend these to inducing points. Counterintuitively, stochastic gradient descent often produces accurate predictions, even in cases where it does not converge quickly to the optimum. We explain this through a spectral characterization of the implicit bias from non-convergence. We show that stochastic gradient descent produces predictive distributions close to the true posterior both in regions with sufficient data coverage, and in regions sufficiently far away from the data. Experimentally, stochastic gradient descent achieves state-of-the-art performance on sufficiently large-scale or ill-conditioned regression tasks. Its uncertainty estimates match the performance of significantly more expensive baselines on a large-scale Bayesian optimization task.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Builds on 28 of its 47 references that are in the map. First posted on arXiv 20 Jun 2023.

Builds on

28 of 47 references.

References · 28

  1. 2016An overview of gradient descent optimization algorithmscited by 1,033
  2. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  3. 2016Automatic chemical design using a data-driven continuous representation of moleculescited by 765
  4. 2018GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Accelerationcited by 609
  5. 2013Gaussian Processes for Big Datacited by 383
  6. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 300
  7. 2014Predictive Entropy Search for Efficient Global Optimization of Black-box Functionscited by 204
  8. 2015Kernel Interpolation for Scalable Structured Gaussian Processes (KISS-GP)cited by 177
  9. 2020Efficiently Sampling Functions from Gaussian Process Posteriorscited by 127
  10. 2019Exact Gaussian Processes on a Million Data Pointscited by 119
  11. 2020Bayesian Deep Learning via Subnetwork Inferencecited by 87
  12. 2015On the Error of Random Fourier Featurescited by 76
16 more
  1. 2017Parallel and Distributed Thompson Sampling for Large-scale Accelerated Exploration of Chemical Spacecited by 76
  2. 2021Benign Overfitting of Constant-Stepsize SGD for Linear Regressioncited by 71
  3. 2020Pathwise Conditioning of Gaussian Processescited by 64
  4. 2015On Sparse variational methods and the Kullback-Leibler divergence between stochastic processescited by 57
  5. 2014Scalable Kernel Methods via Doubly Stochastic Gradientscited by 36
  6. 2016Preconditioning Kernel Matricescited by 33
  7. 2022Adapting the Linearised Laplace Model Evidence for Modern Deep Learningcited by 29
  8. 2021Tighter Bounds on the Log Marginal Likelihood of Gaussian Process Regression Using Conjugate Gradientscited by 20
  9. 2022Sampling-based inference for large linear models, with application to linearised Laplacecited by 18
  10. 2021Connections and Equivalences between the Nystr"om Method and Sparse Variational Gaussian Processescited by 17
  11. 2020State Space Expectation Propagation: Efficient Inference Schemes for Temporal Gaussian Processescited by 10
  12. 2022Uncertainty Estimation for Computed Tomography with a Linearised Deep Image Priorcited by 10
  13. 2022Bayesian Experimental Design for Computed Tomography with the Linearised Deep Image Priorcited by 7
  14. 2022Numerically Stable Sparse Gaussian Processes via Minimum Separation using Cover Treescited by 5
  15. 2015Enabling scalable stochastic gradient-based inference for Gaussian processes by employing the Unbiased LInear System SolvEr (ULISSE)cited by 3
  16. 2023On the Convergence of Stochastic Gradient Descent for Linear Inverse Problems in Banach Spacescited by 2

Built on by

Cited by year

Citing papers per year: 2023: 7; 2024: 14; 2025: 6.

27 papers.

Sort:

  1. 2023A General Framework for User-Guided Bayesian Optimizationcited by 22
  2. 2023Stochastic Gradient Descent for Gaussian Processes Done Rightcited by 17
  3. 2023Large-Scale Gaussian Processes via Alternating Projectioncited by 11
  4. 2024Scaling Gaussian Processes for Learning Curve Prediction via Latent Kronecker Structurecited by 10
  5. 2023A Unifying Perspective on Non-Stationary Kernels for Deeper Gaussian Processescited by 8
  6. 2023Early-Exit Neural Networks with Nested Prediction Setscited by 6
  7. 2024Towards safe and tractable Gaussian process-based MPC: Efficient sampling within a sequential quadratic programming frameworkcited by 6
  8. 2024Improving Linear System Solvers for Hyperparameter Optimisation in Iterative Gaussian Processescited by 5
  9. 2024Towards Exact Computation of Inductive Biascited by 5
  10. 2024Optimizing Posterior Samples for Bayesian Optimization via Rootfindingcited by 5
  11. 2025Scalable Gaussian Processes with Latent Kronecker Structurecited by 5
  12. 2025Graph Random Features for Scalable Gaussian Processescited by 5
15 more
  1. 2024A Quadrature Approach for General-Purpose Batch Bayesian Optimization via Probabilistic Liftingcited by 3
  2. 2024Towards Gaussian Process for operator learning: an uncertainty aware resolution independent operator learning algorithm for computational mechanicscited by 3
  3. 2025Turbocharging Gaussian Process Inference with Approximate Sketch-and-Projectcited by 3
  4. 2023Online Laplace Model Selection Revisitedcited by 2
  5. 2023Trigonometric Quadrature Fourier Features for Scalable Gaussian Process Regressioncited by 2
  6. 2024Recommendations for Baselines and Benchmarking Approximate Gaussian Processescited by 2
  7. 2024Active Few-Shot Fine-Tuningcited by 2
  8. 2024Fixed-Mean Gaussian Processes for Post-hoc Bayesian Deep Learningcited by 2
  9. 2025Stochastic Poisson Surface Reconstruction with One Solve using Geometric Gaussian Processescited by 2
  10. 2024Warm Start Marginal Likelihood Optimisation for Iterative Gaussian Processescited by 1
  11. 2024Gaussian Processes Sampling with Sparse Grids under Additive Schwarz Preconditionercited by 0
  12. 2024Aggregation Models with Optimal Weights for Distributed Gaussian Processescited by 0
  13. 2024Exploiting Hankel-Toeplitz Structures for Fast Computation of Kernel Precision Matricescited by 0
  14. 2025Variational Deep Learning via Implicit Regularizationcited by 0
  15. 2025Warm-Starting Iterative Gaussian Processes for Faster Sequential Inferencecited by 0