# Incremental Ensemble Gaussian Processes

arXiv: https://arxiv.org/abs/2110.06777 · HTML: https://mlsanity.com/paper/2110.06777/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 68 references that are in the map.
- First posted on arXiv 13 Oct 2021.

## Builds on (most cited first, 16 of 16)

- [Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs](https://mlsanity.com/paper/1603.09320/) (2016)
- [Introduction to Online Convex Optimization](https://mlsanity.com/paper/1909.05207/) (2019)
- [GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration](https://mlsanity.com/paper/1809.11165/) (2018)
- [Exact Gaussian Processes on a Million Data Points](https://mlsanity.com/paper/1903.08114/) (2019)
- [Structure Discovery in Nonparametric Regression through Compositional Kernel Search](https://mlsanity.com/paper/1302.4922/) (2013)
- [Distributed Gaussian Processes](https://mlsanity.com/paper/1502.02843/) (2015)
- [Streaming Sparse Gaussian Process Approximations](https://mlsanity.com/paper/1705.07131/) (2017)
- [Preconditioning Kernel Matrices](https://mlsanity.com/paper/1602.06693/) (2016)
- [Random Feature-based Online Multi-kernel Learning in Environments with Unknown Dynamics](https://mlsanity.com/paper/1712.09983/) (2017)
- [Scaling up the Automatic Statistician: Scalable Structure Discovery using Gaussian Processes](https://mlsanity.com/paper/1706.02524/) (2017)
- [Kernel Interpolation for Scalable Online Gaussian Processes](https://mlsanity.com/paper/2103.01454/) (2021)
- [Deep Structured Mixtures of Gaussian Processes](https://mlsanity.com/paper/1910.04536/) (2019)
- [Improving the Gaussian Process Sparse Spectrum Approximation by Representing Uncertainty in Frequency Inputs](https://mlsanity.com/paper/1503.02424/) (2015)
- [Streaming Kernel Principal Component Analysis](https://mlsanity.com/paper/1512.05059/) (2015)
- [GP-Localize: Persistent Mobile Robot Localization using Online Sparse Gaussian Process Observation Model](https://mlsanity.com/paper/1404.5165/) (2014)
- [Scalable Bayesian Non-linear Matrix Completion](https://mlsanity.com/paper/1908.01009/) (2019)

## Built on by (most cited first, 15 of 15)

- [Surrogate modeling for Bayesian optimization beyond a single Gaussian process](https://mlsanity.com/paper/2205.14090/) (2022)
- [Dynamic Online Ensembles of Basis Expansions](https://mlsanity.com/paper/2405.01365/) (2024)
- [Robust and Adaptive Temporal-Difference Learning Using An Ensemble of Gaussian Processes](https://mlsanity.com/paper/2112.00882/) (2021)
- [Online scalable Gaussian processes with conformal prediction for guaranteed coverage](https://mlsanity.com/paper/2410.05444/) (2024)
- [Weighted Ensembles for Active Learning with Adaptivity](https://mlsanity.com/paper/2206.05009/) (2022)
- [A Gaussian Process-based Streaming Algorithm for Prediction of Time Series With Regimes and Outliers](https://mlsanity.com/paper/2406.00570/) (2024)
- [Fine-tuning LLMs with variational Bayesian last layer for high-dimensional Bayesian optimization](https://mlsanity.com/paper/2510.01471/) (2025)
- [A Trainable Approach to Zero-delay Smoothing Spline Interpolation](https://mlsanity.com/paper/2203.03776/) (2022)
- [Decentralized Online Ensembles of Gaussian Processes for Multi-Agent Systems](https://mlsanity.com/paper/2502.05301/) (2025)
- [Physics Inspired Approaches To Understanding Gaussian Processes](https://mlsanity.com/paper/2305.10748/) (2023)
- [Incremental Structure Discovery of Classification via Sequential Monte Carlo](https://mlsanity.com/paper/2408.07875/) (2024)
- [Recursive Gaussian Process State Space Model](https://mlsanity.com/paper/2411.14679/) (2024)
- [A Novel Multi-Criteria Local Latin Hypercube Refinement System for Commutation Angle Improvement in IPMSMs](https://mlsanity.com/paper/2503.03372/) (2025)
- [Bayesian Ensembling: Insights from Online Optimization and Empirical Bayes](https://mlsanity.com/paper/2505.15638/) (2025)
- [Bayesian optimization with kernel ensembles and disagreement-based acquisition for source localization and acoustic inversion](https://mlsanity.com/paper/2609.14262/) (2026)

## Abstract (arXiv)

Belonging to the family of Bayesian nonparametrics, Gaussian process (GP) based approaches have well-documented merits not only in learning over a rich class of nonlinear functions, but also in quantifying the associated uncertainty. However, most GP methods rely on a single preselected kernel function, which may fall short in characterizing data samples that arrive sequentially in time-critical applications. To enable it online kernel adaptation, the present work advocates an incremental ensemble (IE-) GP framework, where an EGP meta-learner employs an it ensemble of GP learners, each having a unique kernel belonging to a prescribed kernel dictionary. With each GP expert leveraging the random feature-based approximation to perform online prediction and model update with it scalability, the EGP meta-learner capitalizes on data-adaptive weights to synthesize the per-expert predictions. Further, the novel IE-GP is generalized to accommodate time-varying functions by modeling structured dynamics at the EGP meta-learner and within each GP learner. To benchmark the performance of IE-GP and its dynamic variant in the adversarial setting where the modeling assumptions are violated, rigorous performance analysis has been conducted via the notion of regret, as the norm in online convex optimization. Last but not the least, online unsupervised learning for dimensionality reduction is explored under the novel IE-GP framework. Synthetic and real data tests demonstrate the effectiveness of the proposed schemes.