# Using stacking to average Bayesian predictive distributions

arXiv: https://arxiv.org/abs/1704.02030 · HTML: https://mlsanity.com/paper/1704.02030/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Apr 2017.

## Built on by (most cited first, 30 of 39)

- [Forecast combinations: an over 50-year review](https://mlsanity.com/paper/2205.04216/) (2022)
- [Bayesian Active Learning for Scanning Probe Microscopy: from Gaussian Processes to Hypothesis Learning](https://mlsanity.com/paper/2205.15458/) (2022)
- [Bayesian hierarchical stacking: Some models are (somewhere) useful](https://mlsanity.com/paper/2101.08954/) (2021)
- [A Tree-based Model Averaging Approach for Personalized Treatment Effect Estimation from Heterogeneous Data Sources](https://mlsanity.com/paper/2103.06261/) (2021)
- [Multiple Importance Sampling ELBO and Deep Ensembles of Variational Approximations](https://mlsanity.com/paper/2202.10951/) (2022)
- [A review of machine learning concepts and methods for addressing challenges in probabilistic hydrological post-processing and forecasting](https://mlsanity.com/paper/2206.08998/) (2022)
- [Simulation-based Bayesian inference under model misspecification](https://mlsanity.com/paper/2503.12315/) (2025)
- [Fast and Scalable Spike and Slab Variable Selection in High-Dimensional Gaussian Processes](https://mlsanity.com/paper/2111.04558/) (2021)
- [The Implications of the No-Free-Lunch Theorems for Meta-induction](https://mlsanity.com/paper/2103.11956/) (2021)
- [Meta-Uncertainty in Bayesian Model Comparison](https://mlsanity.com/paper/2210.07278/) (2022)
- [Dynamic Online Ensembles of Basis Expansions](https://mlsanity.com/paper/2405.01365/) (2024)
- [Ensemble Survival Analysis for Preclinical Cognitive Decline Prediction in Alzheimer's Disease Using Longitudinal Biomarkers](https://mlsanity.com/paper/2503.16645/) (2025)
- [Ensemble learning for blending gridded satellite and gauge-measured precipitation data](https://mlsanity.com/paper/2307.06840/) (2023)
- [Simulation-based stacking](https://mlsanity.com/paper/2310.17009/) (2023)
- [Ensemble learning for uncertainty estimation with application to the correction of satellite precipitation products](https://mlsanity.com/paper/2403.10567/) (2024)
- [Stability of Mean-Field Variational Inference](https://mlsanity.com/paper/2506.07856/) (2025)
- [Local Bayesian Dirichlet mixing of imperfect models](https://mlsanity.com/paper/2311.01596/) (2023)
- [FIRE: Multi-fidelity Regression with Distribution-conditioned In-context Learning using Tabular Foundation Models](https://mlsanity.com/paper/2601.22371/) (2026)
- [A Subspace-based Approach for Dimensionality Reduction and Important Variable Selection](https://mlsanity.com/paper/2106.01584/) (2021)
- [Fusion of Gaussian Processes Predictions with Monte Carlo Sampling](https://mlsanity.com/paper/2403.01389/) (2024)
- [Model orthogonalization and Bayesian forecast mixing via Principal Component Analysis](https://mlsanity.com/paper/2405.10839/) (2024)
- [Towards a Probabilistic Fusion Approach for Robust Battery Prognostics](https://mlsanity.com/paper/2405.15292/) (2024)
- [MODL: Multilearner Online Deep Learning](https://mlsanity.com/paper/2405.18281/) (2024)
- [Understanding the Trade-offs in Accuracy and Uncertainty Quantification: Architecture and Inference Choices in Bayesian Neural Networks](https://mlsanity.com/paper/2503.11808/) (2025)
- [Bayesian Model Averaging for Data Driven Decision Making when Causality is Partially Known](https://mlsanity.com/paper/2105.05395/) (2021)
- [Multi-Model Probabilistic Programming](https://mlsanity.com/paper/2208.06329/) (2022)
- [Robust leave-one-out cross-validation for high-dimensional Bayesian models](https://mlsanity.com/paper/2209.09190/) (2022)
- [Bayesian Analysis for Over-parameterized Linear Model via Effective Spectra](https://mlsanity.com/paper/2305.15754/) (2023)
- [Automating Model Comparison in Factor Graphs](https://mlsanity.com/paper/2306.05965/) (2023)
- [On the meaning of uncertainty for ethical AI: philosophy and practice](https://mlsanity.com/paper/2309.05529/) (2023)

## Abstract (arXiv)

The widely recommended procedure of Bayesian model averaging is flawed in the M-open setting in which the true data-generating process is not one of the candidate models being fit. We take the idea of stacking from the point estimation literature and generalize to the combination of predictive distributions, extending the utility function to any proper scoring rule, using Pareto smoothed importance sampling to efficiently compute the required leave-one-out posterior distributions and regularization to get more stability. We compare stacking of predictive distributions to several alternatives: stacking of means, Bayesian model averaging (BMA), pseudo-BMA using AIC-type weighting, and a variant of pseudo-BMA that is stabilized using the Bayesian bootstrap. Based on simulations and real-data applications, we recommend stacking of predictive distributions, with BB-pseudo-BMA as an approximate alternative when computation cost is an issue.