# Stacking for Non-mixing Bayesian Computations: The Curse and Blessing of Multimodal Posteriors

arXiv: https://arxiv.org/abs/2006.12335 · HTML: https://mlsanity.com/paper/2006.12335/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Jun 2020.

## Built on by (most cited first, 29 of 29)

- [Vanilla Bayesian Optimization Performs Great in High Dimensions](https://mlsanity.com/paper/2402.02229/) (2024)
- [Particle algorithms for maximum likelihood training of latent variable models](https://mlsanity.com/paper/2204.12965/) (2022)
- [Bayesian Active Learning with Fully Bayesian Gaussian Processes](https://mlsanity.com/paper/2205.10186/) (2022)
- [Self-Correcting Bayesian Optimization through Bayesian Active Learning](https://mlsanity.com/paper/2304.11005/) (2023)
- [On Stein Variational Neural Network Ensembles](https://mlsanity.com/paper/2106.10760/) (2021)
- [Neural Variational Gradient Descent](https://mlsanity.com/paper/2107.10731/) (2021)
- [Bayesian hierarchical stacking: Some models are (somewhere) useful](https://mlsanity.com/paper/2101.08954/) (2021)
- [Uncertainty Quantification for Machine Learning in Healthcare: A Survey](https://mlsanity.com/paper/2505.02874/) (2025)
- [Tuning-Free Maximum Likelihood Training of Latent Variable Models via Coin Betting](https://mlsanity.com/paper/2305.14916/) (2023)
- [Improving the evaluation of samplers on multi-modal targets](https://mlsanity.com/paper/2504.08916/) (2025)
- [Proximal Interacting Particle Langevin Algorithms](https://mlsanity.com/paper/2406.14292/) (2024)
- [Foundation Posteriors for Approximate Probabilistic Inference](https://mlsanity.com/paper/2205.09735/) (2022)
- [Bernstein Flows for Flexible Posteriors in Variational Bayes](https://mlsanity.com/paper/2202.05650/) (2022)
- [Fast post-process Bayesian inference with Variational Sparse Bayesian Quadrature](https://mlsanity.com/paper/2303.05263/) (2023)
- [Simulation-based stacking](https://mlsanity.com/paper/2310.17009/) (2023)
- [On Uncertainty Quantification for Near-Bayes Optimal Algorithms](https://mlsanity.com/paper/2403.19381/) (2024)
- [Learning Latent Variable Models via Jarzynski-adjusted Langevin Algorithm](https://mlsanity.com/paper/2505.18427/) (2025)
- [Symmetry-driven embedding of networks in hyperbolic space](https://mlsanity.com/paper/2406.10711/) (2024)
- [Understanding the Trade-offs in Accuracy and Uncertainty Quantification: Architecture and Inference Choices in Bayesian Neural Networks](https://mlsanity.com/paper/2503.11808/) (2025)
- [Differentiable neural network representation of multi-well, locally-convex potentials](https://mlsanity.com/paper/2506.17242/) (2025)
- [Position: The Time for Sampling Is Now! Charting a New Course for Bayesian Deep Learning](https://mlsanity.com/paper/2605.21765/) (2026)
- [Transformation Models for Flexible Posteriors in Variational Bayes](https://mlsanity.com/paper/2106.00528/) (2021)
- [Robust scalable initialization for Bayesian variational inference with multi-modal Laplace approximations](https://mlsanity.com/paper/2307.06424/) (2023)
- [On the meaning of uncertainty for ethical AI: philosophy and practice](https://mlsanity.com/paper/2309.05529/) (2023)
- [BayesBlend: Easy Model Blending using Pseudo-Bayesian Model Averaging, Stacking and Hierarchical Stacking in Python](https://mlsanity.com/paper/2405.00158/) (2024)
- [Robust Inference of Dynamic Covariance Using Wishart Processes and Sequential Monte Carlo](https://mlsanity.com/paper/2406.04796/) (2024)
- [Variational Bayesian Bow tie Neural Networks with Shrinkage](https://mlsanity.com/paper/2411.11132/) (2024)
- [Measuring the right thing: justifying metrics in AI impact assessments](https://mlsanity.com/paper/2504.05007/) (2025)
- [Bayesian Federated Cause-of-Death Classification and Quantification Under Distribution Shift](https://mlsanity.com/paper/2505.02257/) (2025)

## Abstract (arXiv)

When working with multimodal Bayesian posterior distributions, Markov chain Monte Carlo (MCMC) algorithms have difficulty moving between modes, and default variational or mode-based approximate inferences will understate posterior uncertainty. And, even if the most important modes can be found, it is difficult to evaluate their relative weights in the posterior. Here we propose an approach using parallel runs of MCMC, variational, or mode-based inference to hit as many modes or separated regions as possible and then combine these using Bayesian stacking, a scalable method for constructing a weighted average of distributions. The result from stacking efficiently samples from multimodal posterior distribution, minimizes cross validation prediction error, and represents the posterior uncertainty better than variational inference, but it is not necessarily equivalent, even asymptotically, to fully Bayesian inference. We present theoretical consistency with an example where the stacked inference approximates the true data generating process from the misspecified model and a non-mixing sampler, from which the predictive performance is better than full Bayesian inference, hence the multimodality can be considered a blessing rather than a curse under model misspecification. We demonstrate practical implementation in several model families: latent Dirichlet allocation, Gaussian process regression, hierarchical regression, horseshoe variable selection, and neural networks.