# Bayesian Ensembling: Insights from Online Optimization and Empirical Bayes

arXiv: https://arxiv.org/abs/2505.15638 · HTML: https://mlsanity.com/paper/2505.15638/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 61 references that are in the map.
- First posted on arXiv 21 May 2025.

## Builds on (most cited first, 14 of 14)

- [Online Learning: A Modern Introduction Using Convex Optimization](https://mlsanity.com/paper/1912.13213/) (2019)
- [Pareto Smoothed Importance Sampling](https://mlsanity.com/paper/1507.02646/) (2015)
- [Using stacking to average Bayesian predictive distributions](https://mlsanity.com/paper/1704.02030/) (2017)
- [Fusion of Probability Density Functions](https://mlsanity.com/paper/2202.11633/) (2022)
- [Incremental Ensemble Gaussian Processes](https://mlsanity.com/paper/2110.06777/) (2021)
- [Bayesian Online Natural Gradient (BONG)](https://mlsanity.com/paper/2405.19681/) (2024)
- [Trading-Off Static and Dynamic Regret in Online Least-Squares and Beyond](https://mlsanity.com/paper/1909.03118/) (2019)
- [Data-Dependent Bounds for Online Portfolio Selection Without Lipschitzness and Smoothness](https://mlsanity.com/paper/2305.13946/) (2023)
- [Soft-Bayes: Prod for Mixtures of Experts with Log-Loss](https://mlsanity.com/paper/1901.02230/) (2019)
- [Online Convex Optimisation: The Optimal Switching Regret for all Segmentations Simultaneously](https://mlsanity.com/paper/2405.20824/) (2024)
- [Dynamic Online Ensembles of Basis Expansions](https://mlsanity.com/paper/2405.01365/) (2024)
- [Bayesian model averaging with the integrated nested Laplace approximation](https://mlsanity.com/paper/1911.00797/) (2019)
- [MODL: Multilearner Online Deep Learning](https://mlsanity.com/paper/2405.18281/) (2024)
- [Decentralized Online Ensembles of Gaussian Processes for Multi-Agent Systems](https://mlsanity.com/paper/2502.05301/) (2025)

## Abstract (arXiv)

We revisit the classical problem of Bayesian ensembles and address the challenge of learning optimal combinations of Bayesian models in an online, continual learning setting. To this end, we reinterpret existing approaches such as Bayesian model averaging (BMA) and Bayesian stacking through a novel empirical Bayes lens, shedding new light on the limitations and pathologies of BMA. Further motivated by insights from online optimization, we propose Online Bayesian Stacking (OBS), a method that optimizes the log-score over predictive distributions to adaptively combine Bayesian models. A key contribution of our work is establishing a novel connection between OBS and portfolio selection, bridging Bayesian ensemble learning with a rich, well-studied theoretical framework that offers efficient algorithms and extensive regret analysis. We further clarify the relationship between OBS and online BMA, showing that they optimize related but distinct cost functions. Through theoretical analysis and empirical evaluation, we identify scenarios where OBS outperforms online BMA and provide principled methods and guidance on when practitioners should prefer one approach over the other.