mlsanityData to 14 Oct 2026

Low-Precision Arithmetic for Fast Gaussian Processes

arXiv 2207.06856 ↗First posted 14 Jul 2022cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

Low-precision arithmetic has had a transformative effect on the training of neural networks, reducing computation, memory and energy requirements. However, despite its promise, low-precision arithmetic has received little attention for Gaussian processes (GPs), largely because GPs require sophisticated linear algebra routines that are unstable in low-precision. We study the different failure modes that can occur when training GPs in half precision. To circumvent these failure modes, we propose a multi-faceted approach involving conjugate gradients with re-orthogonalization, mixed precision, and preconditioning. Our approach significantly improves the numerical stability and practical performance of conjugate gradients in low-precision over a wide range of settings, enabling GPs to train on 1.8 million data points in 10 hours on a single GPU, without any sparse approximations.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 21 of its 54 references that are in the map. First posted on arXiv 14 Jul 2022.

Builds on

21 of 54 references.

References · 21

  1. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  2. 2018GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Accelerationcited by 609
  3. 2013Gaussian Processes for Big Datacited by 383
  4. 2015Deep Learning with Limited Numerical Precisioncited by 376
  5. 2019Exact Gaussian Processes on a Million Data Pointscited by 119
  6. 2020Kernel Operations on the GPU, with Autodiff, without Memory Overflowscited by 104
  7. 2020Kernel methods through the roof: handling billions of points efficientlycited by 89
  8. 2014Fastfood: Approximate Kernel Expansions in Loglinear Timecited by 74
  9. 2020Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisitedcited by 51
  10. 2018Constant-Time Predictive Distributions for Gaussian Processescited by 48
  11. 2018Mixed Precision Training of Convolutional Neural Networks using Integer Operationscited by 45
  12. 2019SWALP : Stochastic Weight Averaging in Low-Precision Trainingcited by 42
9 more
  1. 2016Preconditioning Kernel Matricescited by 33
  2. 2020Fast Matrix Square Roots with Applications to Gaussian Processes and Bayesian Optimizationcited by 26
  3. 2014A la Carte - Learning Fast Kernelscited by 18
  4. 2020Revisiting BFloat16 Trainingcited by 17
  5. 2021When are Iterative Gaussian Processes Reliably Accurate?cited by 12
  6. 2018Low-Precision Random Fourier Features for Memory-Constrained Kernel Approximationcited by 7
  7. 2021Quantization Algorithms for Random Fourier Featurescited by 7
  8. 2013Parallel Gaussian Process Regression with Low-Rank Covariance Matrix Approximationscited by 5
  9. 2018Minimizing convex quadratic with variable precision conjugate gradientscited by 2

Built on by

Cited by year

Citing papers per year: 2023: 5; 2024: 3; 2025: 2.

10 papers.

  1. 2023Large-Scale Gaussian Processes via Alternating Projectioncited by 11
  2. 2023Kernel Regression with Infinite-Width Neural Networks on Millions of Examplescited by 9
  3. 2023CoLA: Exploiting Compositional Structure for Automatic and Efficient Numerical Linear Algebracited by 7
  4. 2023Convolutional Deep Kernel Machinescited by 3
  5. 2025VIKING: Deep variational inference with stochastic projectionscited by 3
  6. 2023Trigonometric Quadrature Fourier Features for Scalable Gaussian Process Regressioncited by 2
  7. 2024Probabilistic Neural Networks (PNNs) for Modeling Aleatoric Uncertainty in Scientific Machine Learningcited by 2
  8. 2024High-Dimensional Gaussian Process Regression with Soft Kernel Interpolationcited by 1
  9. 2025Scaling Gaussian Process Regression with Full Derivative Observationscited by 1
  10. 2024Stochastic Kernel Regularisation Improves Generalisation in Deep Kernel Machinescited by 0