# Gaussian Process Boosting

arXiv: https://arxiv.org/abs/2004.02653 · HTML: https://mlsanity.com/paper/2004.02653/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Apr 2020.

## Built on by (most cited first, 26 of 26)

- [Neural networks for geospatial data](https://mlsanity.com/paper/2304.09157/) (2023)
- [Latent Gaussian Model Boosting](https://mlsanity.com/paper/2105.08966/) (2021)
- [Machine Learning with High-Cardinality Categorical Features in Actuarial Applications](https://mlsanity.com/paper/2301.12710/) (2023)
- [Iterative Methods for Vecchia-Laplace Approximations for Latent Gaussian Process Models](https://mlsanity.com/paper/2310.12000/) (2023)
- [Towards a Unified Framework for Uncertainty-aware Nonlinear Variable Selection with Theoretical Guarantees](https://mlsanity.com/paper/2204.07293/) (2022)
- [Iterative Methods for Full-Scale Gaussian Process Approximations for Large Spatial Data](https://mlsanity.com/paper/2405.14492/) (2024)
- [Enabling Mixed Effects Neural Networks for Diverse, Clustered Data Using Monte Carlo Methods](https://mlsanity.com/paper/2407.01115/) (2024)
- [A Comparison of Machine Learning Methods for Data with High-Cardinality Categorical Variables](https://mlsanity.com/paper/2307.02071/) (2023)
- [A Spatio-Temporal Machine Learning Model for Mortgage Credit Risk: Default Probabilities and Loan Portfolios](https://mlsanity.com/paper/2410.02846/) (2024)
- [Gradient Boosting Reinforcement Learning](https://mlsanity.com/paper/2407.08250/) (2024)
- [Selecting Hyperparameters for Tree-Boosting](https://mlsanity.com/paper/2602.05786/) (2026)
- [A Gradient Boosted Mixed-Model Machine Learning Framework for Vessel Speed in the U.S. Arctic](https://mlsanity.com/paper/2602.12292/) (2026)
- [MFAI: A Scalable Bayesian Matrix Factorization Approach to Leveraging Auxiliary Information](https://mlsanity.com/paper/2303.02566/) (2023)
- [Chasing the Timber Trail: Machine Learning to Reveal Harvest Location Misrepresentation](https://mlsanity.com/paper/2502.14115/) (2025)
- [Scalable Krylov Subspace Methods for Generalized Mixed-Effects Models with Crossed Random Effects](https://mlsanity.com/paper/2505.09552/) (2025)
- [Vecchia-Inducing-Points Full-Scale Approximations for Gaussian Processes](https://mlsanity.com/paper/2507.05064/) (2025)
- [Fast Gaussian process inference by exact Mat'ern kernel decomposition](https://mlsanity.com/paper/2508.01864/) (2025)
- [Gradient Boosted Mixed Models: Flexible Estimation of Mean and Variance Components for Clustered Data](https://mlsanity.com/paper/2511.00217/) (2025)
- [Global Interpretability via Automated Preprocessing: A Framework Inspired by Psychiatric Questionnaires](https://mlsanity.com/paper/2602.23459/) (2026)
- [Random-Effects Algorithm for Random Objects in Metric Spaces](https://mlsanity.com/paper/2605.02693/) (2026)
- [A Censored Transformed Model for Proportional Outcomes with Boundary Mass and an Application to Loss Given Default Modeling](https://mlsanity.com/paper/2606.21515/) (2026)
- [Spectrally Deconfounded Gradient Boosting](https://mlsanity.com/paper/2607.09371/) (2026)
- [Covariance-Boosted Gaussian Processes for Spatiotemporal Irregularities](https://mlsanity.com/paper/2607.23018/) (2026)
- [NPBoost: Neural Processes with Gradient-Boosted Fixed Effects](https://mlsanity.com/paper/2609.28122/) (2026)
- [Graph-Split Bayesian Causal Forest for Spatial Heterogeneous Treatment Effect Estimation](https://mlsanity.com/paper/2609.36046/) (2026)
- [TreeWalker: Partial Evaluation for Grouped Tree-Ensemble Inference](https://mlsanity.com/paper/2610.03939/) (2026)

## Abstract (arXiv)

We introduce a novel way to combine boosting with Gaussian process and mixed effects models. This allows for relaxing, first, the zero or linearity assumption for the prior mean function in Gaussian process and grouped random effects models in a flexible non-parametric way and, second, the independence assumption made in most boosting algorithms. The former is advantageous for prediction accuracy and for avoiding model misspecifications. The latter is important for efficient learning of the fixed effects predictor function and for obtaining probabilistic predictions. Our proposed algorithm is also a novel solution for handling high-cardinality categorical variables in tree-boosting. In addition, we present an extension that scales to large data using a Vecchia approximation for the Gaussian process model relying on novel results for covariance parameter inference. We obtain increased prediction accuracy compared to existing approaches on multiple simulated and real-world data sets.