# Robust Gaussian Processes via Relevance Pursuit

arXiv: https://arxiv.org/abs/2410.24222 · HTML: https://mlsanity.com/paper/2410.24222/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 66 references that are in the map.
- First posted on arXiv 31 Oct 2024.

## Builds on (most cited first, 22 of 22)

- [A Tutorial on Bayesian Optimization](https://mlsanity.com/paper/1807.02811/) (2018)
- [GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration](https://mlsanity.com/paper/1809.11165/) (2018)
- [Unexpected Improvements to Expected Improvement for Bayesian Optimization](https://mlsanity.com/paper/2310.20708/) (2023)
- [Batched Large-scale Bayesian Optimization in High-dimensional Spaces](https://mlsanity.com/paper/1706.01445/) (2017)
- [Restricted Strong Convexity Implies Weak Submodularity](https://mlsanity.com/paper/1612.00804/) (2016)
- [Student-t Processes as Alternatives to Gaussian Processes](https://mlsanity.com/paper/1402.4306/) (2014)
- [Scalable Log Determinants for Gaussian Process Kernel Learning](https://mlsanity.com/paper/1711.03481/) (2017)
- [Discovering Many Diverse Solutions with Bayesian Optimization](https://mlsanity.com/paper/2210.10953/) (2022)
- [Bayesian Active Learning with Fully Bayesian Gaussian Processes](https://mlsanity.com/paper/2205.10186/) (2022)
- [Autonomous synthesis of metastable materials](https://mlsanity.com/paper/2101.07385/) (2021)
- [Self-Correcting Bayesian Optimization through Bayesian Active Learning](https://mlsanity.com/paper/2304.11005/) (2023)
- [Robust and Conjugate Gaussian Process Regression](https://mlsanity.com/paper/2311.00463/) (2023)
- [A Unified Optimization View on Generalized Matching Pursuit and Frank-Wolfe](https://mlsanity.com/paper/1702.06457/) (2017)
- [Scaling Gaussian Processes for Learning Curve Prediction via Latent Kronecker Structure](https://mlsanity.com/paper/2410.09239/) (2024)
- [On Matching Pursuit and Coordinate Descent](https://mlsanity.com/paper/1803.09539/) (2018)
- [Sparse Bayesian Learning via Stepwise Regression](https://mlsanity.com/paper/2106.06095/) (2021)
- [Robust Gaussian Process Regression with Huber Likelihood](https://mlsanity.com/paper/2301.07858/) (2023)
- [Robust Gaussian Process Regression with a Bias Model](https://mlsanity.com/paper/2001.04639/) (2020)
- [Robust Bayesian Optimization with Student-t Likelihood](https://mlsanity.com/paper/1707.05729/) (2017)
- [Robust Gaussian Process Regression Based on Iterative Trimming](https://mlsanity.com/paper/2011.11057/) (2020)
- [On the Optimality of Backward Regression: Sparse Recovery and Subset Selection](https://mlsanity.com/paper/2106.03235/) (2021)
- [Performance guarantees of forward and reverse greedy algorithms for minimizing nonsupermodular nonsubmodular functions on a matroid](https://mlsanity.com/paper/2103.01135/) (2021)

## Built on by (most cited first, 8 of 8)

- [Robust and Conjugate Spatio-Temporal Gaussian Processes](https://mlsanity.com/paper/2502.02450/) (2025)
- [HiGP: A high-performance Python package for Gaussian Process](https://mlsanity.com/paper/2503.02259/) (2025)
- [Multi-Output Robust and Conjugate Gaussian Processes](https://mlsanity.com/paper/2510.26401/) (2025)
- [Robust and Computation-Aware Gaussian Processes](https://mlsanity.com/paper/2505.21133/) (2025)
- [Learning Rate Should Scale Inversely with High-Order Data Moments in High-Dimensional Online Independent Component Analysis](https://mlsanity.com/paper/2509.15127/) (2025)
- [ORTHOBO: Orthogonal Bayesian Hyperparameter Optimization](https://mlsanity.com/paper/2605.06454/) (2026)
- [Joint Model and Data Sparsification via the Marginal Likelihood](https://mlsanity.com/paper/2605.29908/) (2026)
- [What Can a Gaussian Process Design Test](https://mlsanity.com/paper/2610.10122/) (2026)

## Abstract (arXiv)

Gaussian processes (GPs) are non-parametric probabilistic regression models that are popular due to their flexibility, data efficiency, and well-calibrated uncertainty estimates. However, standard GP models assume homoskedastic Gaussian noise, while many real-world applications are subject to non-Gaussian corruptions. Variants of GPs that are more robust to alternative noise models have been proposed, and entail significant trade-offs between accuracy and robustness, and between computational requirements and theoretical guarantees. In this work, we propose and study a GP model that achieves robustness against sparse outliers by inferring data-point-specific noise levels with a sequential selection procedure maximizing the log marginal likelihood that we refer to as relevance pursuit. We show, surprisingly, that the model can be parameterized such that the associated log marginal likelihood is strongly concave in the data-point-specific noise variances, a property rarely found in either robust regression objectives or GP marginal likelihoods. This in turn implies the weak submodularity of the corresponding subset selection problem, and thereby proves approximation guarantees for the proposed algorithm. We compare the model's performance relative to other approaches on diverse regression and Bayesian optimization tasks, including the challenging but common setting of sparse corruptions of the labels within or close to the function range.