# Bayesian hierarchical stacking: Some models are (somewhere) useful

arXiv: https://arxiv.org/abs/2101.08954 · HTML: https://mlsanity.com/paper/2101.08954/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 53 references that are in the map.
- First posted on arXiv 22 Jan 2021.

## Builds on (most cited first, 13 of 13)

- [Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC](https://mlsanity.com/paper/1507.04544/) (2015)
- [Bayesian Workflow](https://mlsanity.com/paper/2011.01808/) (2020)
- [Pareto Smoothed Importance Sampling](https://mlsanity.com/paper/1507.02646/) (2015)
- [Using stacking to average Bayesian predictive distributions](https://mlsanity.com/paper/1704.02030/) (2017)
- [Stacking Models for Nearly Optimal Link Prediction in Complex Networks](https://mlsanity.com/paper/1909.07578/) (2019)
- [Stacking for Non-mixing Bayesian Computations: The Curse and Blessing of Multimodal Posteriors](https://mlsanity.com/paper/2006.12335/) (2020)
- [Comparison of Bayesian predictive methods for model selection](https://mlsanity.com/paper/1503.08650/) (2015)
- [Approximate leave-future-out cross-validation for Bayesian time series models](https://mlsanity.com/paper/1902.06281/) (2019)
- [Improved prediction accuracy for disease risk mapping using Gaussian Process stacked generalisation](https://mlsanity.com/paper/1612.03278/) (2016)
- [Minimax Optimal Bayesian Aggregation](https://mlsanity.com/paper/1403.1345/) (2014)
- [Testing hypotheses via a mixture estimation model](https://mlsanity.com/paper/1412.2044/) (2014)
- [Reducing the error of Monte Carlo Algorithms by Learning Control Variates](https://mlsanity.com/paper/1606.02261/) (2016)
- [Bayesian Aggregation](https://mlsanity.com/paper/1912.11218/) (2019)

## Built on by (most cited first, 14 of 14)

- [Amortized Bayesian Multilevel Models](https://mlsanity.com/paper/2408.13230/) (2024)
- [Dynamic Online Ensembles of Basis Expansions](https://mlsanity.com/paper/2405.01365/) (2024)
- [Local Bayesian Dirichlet mixing of imperfect models](https://mlsanity.com/paper/2311.01596/) (2023)
- [Fusion of Gaussian Processes Predictions with Monte Carlo Sampling](https://mlsanity.com/paper/2403.01389/) (2024)
- [Model orthogonalization and Bayesian forecast mixing via Principal Component Analysis](https://mlsanity.com/paper/2405.10839/) (2024)
- [Fitting Low-rank Models on Egocentrically Sampled Partial Networks](https://mlsanity.com/paper/2303.11230/) (2023)
- [Locking and Quacking: Stacking Bayesian model predictions by log-pooling and superposition](https://mlsanity.com/paper/2305.07334/) (2023)
- [Automating Model Comparison in Factor Graphs](https://mlsanity.com/paper/2306.05965/) (2023)
- [On the meaning of uncertainty for ethical AI: philosophy and practice](https://mlsanity.com/paper/2309.05529/) (2023)
- [Beyond Bayesian Model Averaging over Paths in Probabilistic Programs with Stochastic Support](https://mlsanity.com/paper/2310.14888/) (2023)
- [BayesBlend: Easy Model Blending using Pseudo-Bayesian Model Averaging, Stacking and Hierarchical Stacking in Python](https://mlsanity.com/paper/2405.00158/) (2024)
- [Input Adaptive Bayesian Model Averaging](https://mlsanity.com/paper/2510.22054/) (2025)
- [A Divergence-Based Method for Weighting and Averaging Model Predictions](https://mlsanity.com/paper/2604.24172/) (2026)
- [A renormalization-group inspired lattice-based framework for piecewise generalized linear models](https://mlsanity.com/paper/2605.05493/) (2026)

## Abstract (arXiv)

Stacking is a widely used model averaging technique that asymptotically yields optimal predictions among linear averages. We show that stacking is most effective when model predictive performance is heterogeneous in inputs, and we can further improve the stacked mixture with a hierarchical model. We generalize stacking to Bayesian hierarchical stacking. The model weights are varying as a function of data, partially-pooled, and inferred using Bayesian inference. We further incorporate discrete and continuous inputs, other structured priors, and time series and longitudinal data. To verify the performance gain of the proposed method, we derive theory bounds, and demonstrate on several applied problems.