# Dynamic Online Ensembles of Basis Expansions

arXiv: https://arxiv.org/abs/2405.01365 · HTML: https://mlsanity.com/paper/2405.01365/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 56 references that are in the map.
- First posted on arXiv 2 May 2024.

## Builds on (most cited first, 18 of 18)

- [Introduction to Online Convex Optimization](https://mlsanity.com/paper/1909.05207/) (2019)
- [Online Learning: A Modern Introduction Using Convex Optimization](https://mlsanity.com/paper/1912.13213/) (2019)
- [Gaussian Process Kernels for Pattern Discovery and Extrapolation](https://mlsanity.com/paper/1302.4245/) (2013)
- [Structure Discovery in Nonparametric Regression through Compositional Kernel Search](https://mlsanity.com/paper/1302.4922/) (2013)
- [Hilbert Space Methods for Reduced-Rank Gaussian Process Regression](https://mlsanity.com/paper/1401.5508/) (2014)
- [Variational Fourier features for Gaussian processes](https://mlsanity.com/paper/1611.06740/) (2016)
- [A survey on high-dimensional Gaussian process modeling with application to Bayesian optimization](https://mlsanity.com/paper/2111.05040/) (2021)
- [On the Error of Random Fourier Features](https://mlsanity.com/paper/1506.02785/) (2015)
- [Streaming Sparse Gaussian Process Approximations](https://mlsanity.com/paper/1705.07131/) (2017)
- [Using stacking to average Bayesian predictive distributions](https://mlsanity.com/paper/1704.02030/) (2017)
- [Marginal likelihood computation for model selection and hypothesis testing: an extensive review](https://mlsanity.com/paper/2005.08334/) (2020)
- [Practical Hilbert space approximate Bayesian Gaussian processes for probabilistic programming](https://mlsanity.com/paper/2004.11408/) (2020)
- [Surrogate modeling for Bayesian optimization beyond a single Gaussian process](https://mlsanity.com/paper/2205.14090/) (2022)
- [Randomly Projected Additive Gaussian Processes for Regression](https://mlsanity.com/paper/1912.12834/) (2019)
- [Kernel Interpolation for Scalable Online Gaussian Processes](https://mlsanity.com/paper/2103.01454/) (2021)
- [Incremental Ensemble Gaussian Processes](https://mlsanity.com/paper/2110.06777/) (2021)
- [Bayesian hierarchical stacking: Some models are (somewhere) useful](https://mlsanity.com/paper/2101.08954/) (2021)
- [Sequential Estimation of Gaussian Process-based Deep State-Space Models](https://mlsanity.com/paper/2301.12528/) (2023)

## Built on by (most cited first, 5 of 5)

- [A Gaussian Process-based Streaming Algorithm for Prediction of Time Series With Regimes and Outliers](https://mlsanity.com/paper/2406.00570/) (2024)
- [Robust, Online, and Adaptive Decentralized Gaussian Processes](https://mlsanity.com/paper/2509.18011/) (2025)
- [Decentralized Online Ensembles of Gaussian Processes for Multi-Agent Systems](https://mlsanity.com/paper/2502.05301/) (2025)
- [SEEK: Self-adaptive Explainable Kernel For Nonstationary Gaussian Processes](https://mlsanity.com/paper/2503.14785/) (2025)
- [Bayesian Ensembling: Insights from Online Optimization and Empirical Bayes](https://mlsanity.com/paper/2505.15638/) (2025)

## Abstract (arXiv)

Practical Bayesian learning often requires (1) online inference, (2) dynamic models, and (3) ensembling over multiple different models. Recent advances have shown how to use random feature approximations to achieve scalable, online ensembling of Gaussian processes with desirable theoretical properties and fruitful applications. One key to these methods' success is the inclusion of a random walk on the model parameters, which makes models dynamic. We show that these methods can be generalized easily to any basis expansion model and that using alternative basis expansions, such as Hilbert space Gaussian processes, often results in better performance. To simplify the process of choosing a specific basis expansion, our method's generality also allows the ensembling of several entirely different models, for example, a Gaussian process and polynomial regression. Finally, we propose a novel method to ensemble static and dynamic models together.