# High-Dimensional Gaussian Process Inference with Derivatives

arXiv: https://arxiv.org/abs/2102.07542 · HTML: https://mlsanity.com/paper/2102.07542/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 45 references that are in the map.
- First posted on arXiv 15 Feb 2021.

## Builds on (most cited first, 20 of 20)

- [GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration](https://mlsanity.com/paper/1809.11165/) (2018)
- [Stochastic Gradient Hamiltonian Monte Carlo](https://mlsanity.com/paper/1402.4102/) (2014)
- [A Conceptual Introduction to Hamiltonian Monte Carlo](https://mlsanity.com/paper/1701.02434/) (2017)
- [Kernel Interpolation for Scalable Structured Gaussian Processes (KISS-GP)](https://mlsanity.com/paper/1503.01057/) (2015)
- [Gaussian Process Kernels for Pattern Discovery and Extrapolation](https://mlsanity.com/paper/1302.4245/) (2013)
- [Hilbert Space Methods for Reduced-Rank Gaussian Process Regression](https://mlsanity.com/paper/1401.5508/) (2014)
- [Probabilistic Numerics and Uncertainty in Computations](https://mlsanity.com/paper/1506.01326/) (2015)
- [Bayesian Optimization with Gradients](https://mlsanity.com/paper/1703.04389/) (2017)
- [Linearly constrained Gaussian processes](https://mlsanity.com/paper/1703.00787/) (2017)
- [Product Kernel Interpolation for Scalable Gaussian Processes](https://mlsanity.com/paper/1802.08903/) (2018)
- [Modeling and interpolation of the ambient magnetic field by Gaussian processes](https://mlsanity.com/paper/1509.04634/) (2015)
- [Scaling Gaussian Process Regression with Derivatives](https://mlsanity.com/paper/1810.12283/) (2018)
- [Probabilistic Interpretation of Linear Solvers](https://mlsanity.com/paper/1402.2058/) (2014)
- [Probabilistic Linear Solvers for Machine Learning](https://mlsanity.com/paper/2010.09691/) (2020)
- [A Bayesian Conjugate Gradient Method](https://mlsanity.com/paper/1801.05242/) (2018)
- [Stochastic quasi-Newton with line-search regularization](https://mlsanity.com/paper/1909.01238/) (2019)
- [Deep Bayesian Quadrature Policy Optimization](https://mlsanity.com/paper/2006.15637/) (2020)
- [On the construction of probabilistic Newton-type algorithms](https://mlsanity.com/paper/1704.01382/) (2017)
- [Active Probabilistic Inference on Matrices for Pre-Conditioning in Stochastic Optimization](https://mlsanity.com/paper/1902.07557/) (2019)
- [SLEIPNIR: Deterministic and Provably Accurate Feature Expansion for Gaussian Process Regression with Derivatives](https://mlsanity.com/paper/2003.02658/) (2020)

## Built on by (most cited first, 16 of 16)

- [Unexpected Improvements to Expected Improvement for Bayesian Optimization](https://mlsanity.com/paper/2310.20708/) (2023)
- [Sparse Cholesky Factorization for Solving Nonlinear PDEs via Gaussian Processes](https://mlsanity.com/paper/2304.01294/) (2023)
- [Bayesian Numerical Methods for Nonlinear Partial Differential Equations](https://mlsanity.com/paper/2104.12587/) (2021)
- [Scaling Gaussian Processes with Derivative Information Using Variational Inference](https://mlsanity.com/paper/2107.04061/) (2021)
- [Monotonicity and Double Descent in Uncertainty Estimation with Gaussian Processes](https://mlsanity.com/paper/2210.07612/) (2022)
- [Scalable First-Order Bayesian Optimization via Structured Automatic Differentiation](https://mlsanity.com/paper/2206.08366/) (2022)
- [Algorithmic Differentiation for Automated Modeling of Machine Learned Force Fields](https://mlsanity.com/paper/2208.12104/) (2022)
- [NeST-BO: Fast Local Bayesian Optimization via Newton-Step Targeting of Gradient and Hessian Information](https://mlsanity.com/paper/2510.05516/) (2025)
- [BayeSQP: Bayesian Optimization through Sequential Quadratic Programming](https://mlsanity.com/paper/2602.03232/) (2026)
- [The Hyperdimensional Transform for Distributional Modelling, Regression and Classification](https://mlsanity.com/paper/2311.08150/) (2023)
- [Random Function Descent](https://mlsanity.com/paper/2305.01377/) (2023)
- [Scaling Gaussian Process Regression with Full Derivative Observations](https://mlsanity.com/paper/2505.09134/) (2025)
- [Integrated Gradient attribution for Gaussian Processes with non-Gaussian likelihoods](https://mlsanity.com/paper/2205.12797/) (2022)
- [Linear cost and exponentially convergent approximation of Gaussian Mat'ern processes on intervals](https://mlsanity.com/paper/2410.13000/) (2024)
- [Novel Pivoted Cholesky Decompositions for Efficient Gaussian Process Inference](https://mlsanity.com/paper/2507.20678/) (2025)
- [Scalable Derivative Gaussian Processes via Exact Gradient Reduction](https://mlsanity.com/paper/2606.02909/) (2026)

## Abstract (arXiv)

Although it is widely known that Gaussian processes can be conditioned on observations of the gradient, this functionality is of limited use due to the prohibitive computational cost of O(N3 D3) in data points N and dimension D. The dilemma of gradient observations is that a single one of them comes at the same cost as D independent function evaluations, so the latter are often preferred. Careful scrutiny reveals, however, that derivative observations give rise to highly structured kernel Gram matrices for very general classes of kernels (inter alia, stationary kernels). We show that in the low-data regime N<D, the Gram matrix can be decomposed in a manner that reduces the cost of inference to O(N2D + (N2)3) (i.e., linear in the number of dimensions) and, in special cases, to O(N2D + N3). This reduction in complexity opens up new use-cases for inference with gradients especially in the high-dimensional regime, where the information-to-cost ratio of gradient observations significantly increases. We demonstrate this potential in a variety of tasks relevant for machine learning, such as optimization and Hamiltonian Monte Carlo with predictive gradients.