# Beyond Bayesian Model Averaging over Paths in Probabilistic Programs with Stochastic Support

arXiv: https://arxiv.org/abs/2310.14888 · HTML: https://mlsanity.com/paper/2310.14888/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 96 references that are in the map.
- First posted on arXiv 23 Oct 2023.

## Builds on (most cited first, 30 of 31)

- [Importance Weighted Autoencoders](https://mlsanity.com/paper/1509.00519/) (2015)
- [Pyro: Deep Universal Probabilistic Programming](https://mlsanity.com/paper/1810.09538/) (2018)
- [Black Box Variational Inference](https://mlsanity.com/paper/1401.0118/) (2013)
- [User-friendly introduction to PAC-Bayes bounds](https://mlsanity.com/paper/2110.11216/) (2021)
- [A Conceptual Introduction to Hamiltonian Monte Carlo](https://mlsanity.com/paper/1701.02434/) (2017)
- [Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC](https://mlsanity.com/paper/1507.04544/) (2015)
- [An Introduction to Probabilistic Programming](https://mlsanity.com/paper/1809.10756/) (2018)
- [Pareto Smoothed Importance Sampling](https://mlsanity.com/paper/1507.02646/) (2015)
- [Using stacking to average Bayesian predictive distributions](https://mlsanity.com/paper/1704.02030/) (2017)
- [Automated Variational Inference in Probabilistic Programming](https://mlsanity.com/paper/1301.1299/) (2013)
- [Inference Compilation and Universal Probabilistic Programming](https://mlsanity.com/paper/1610.09900/) (2016)
- [Deep Probabilistic Programming](https://mlsanity.com/paper/1701.03757/) (2017)
- [A New Approach to Probabilistic Programming Inference](https://mlsanity.com/paper/1507.00996/) (2015)
- [Venture: a higher-order probabilistic programming platform with programmable inference](https://mlsanity.com/paper/1404.0099/) (2014)
- [Automatic Differentiation Variational Inference with Mixtures](https://mlsanity.com/paper/2003.01687/) (2020)
- [Layered Adaptive Importance Sampling](https://mlsanity.com/paper/1505.04732/) (2015)
- [Etalumis: Bringing Probabilistic Programming to Scientific Simulators at Scale](https://mlsanity.com/paper/1907.03382/) (2019)
- [Design and Implementation of Probabilistic Programming Language Anglican](https://mlsanity.com/paper/1608.05263/) (2016)
- [Bayesian hierarchical stacking: Some models are (somewhere) useful](https://mlsanity.com/paper/2101.08954/) (2021)
- [A Lambda-Calculus Foundation for Universal Probabilistic Programming](https://mlsanity.com/paper/1512.08990/) (2015)
- [Semantics for probabilistic programming: higher-order functions, continuous distributions, and soft constraints](https://mlsanity.com/paper/1601.04943/) (2016)
- [ADEV: Sound Automatic Differentiation of Expected Values of Probabilistic Programs](https://mlsanity.com/paper/2212.06386/) (2022)
- [Automated learning with a probabilistic programming language: Birch](https://mlsanity.com/paper/1810.01539/) (2018)
- [Divide, Conquer, and Combine: a New Inference Strategy for Probabilistic Programs with Stochastic Support](https://mlsanity.com/paper/1910.13324/) (2019)
- [Holes in Bayesian Statistics](https://mlsanity.com/paper/2002.06467/) (2020)
- [Reproducible Model Selection Using Bagged Posteriors](https://mlsanity.com/paper/2007.14845/) (2020)
- [Interacting Particle Markov Chain Monte Carlo](https://mlsanity.com/paper/1602.05128/) (2016)
- [Composing inference algorithms as program transformations](https://mlsanity.com/paper/1603.01882/) (2016)
- [Attention for Inference Compilation](https://mlsanity.com/paper/1910.11961/) (2019)
- [Symbolic Parallel Adaptive Importance Sampling for Probabilistic Program Analysis](https://mlsanity.com/paper/2010.05050/) (2020)

## Abstract (arXiv)

The posterior in probabilistic programs with stochastic support decomposes as a weighted sum of the local posterior distributions associated with each possible program path. We show that making predictions with this full posterior implicitly performs a Bayesian model averaging (BMA) over paths. This is potentially problematic, as BMA weights can be unstable due to model misspecification or inference approximations, leading to sub-optimal predictions in turn. To remedy this issue, we propose alternative mechanisms for path weighting: one based on stacking and one based on ideas from PAC-Bayes. We show how both can be implemented as a cheap post-processing step on top of existing inference engines. In our experiments, we find them to be more robust and lead to better predictions compared to the default BMA weights.