# Attention for Inference Compilation

arXiv: https://arxiv.org/abs/1910.11961 · HTML: https://mlsanity.com/paper/1910.11961/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Oct 2019.

## Built on by (most cited first, 3 of 3)

- [Rethinking Variational Inference for Probabilistic Programs with Stochastic Support](https://mlsanity.com/paper/2311.00594/) (2023)
- [Beyond Bayesian Model Averaging over Paths in Probabilistic Programs with Stochastic Support](https://mlsanity.com/paper/2310.14888/) (2023)
- [Semi-Amortized Bayesian Inference with Reusable Local Posteriors](https://mlsanity.com/paper/2610.18945/) (2026)

## Abstract (arXiv)

We present a new approach to automatic amortized inference in universal probabilistic programs which improves performance compared to current methods. Our approach is a variation of inference compilation (IC) which leverages deep neural networks to approximate a posterior distribution over latent variables in a probabilistic program. A challenge with existing IC network architectures is that they can fail to model long-range dependencies between latent variables. To address this, we introduce an attention mechanism that attends to the most salient variables previously sampled in the execution of a probabilistic program. We demonstrate that the addition of attention allows the proposal distributions to better match the true posterior, enhancing inference about latent variables in simulators.