# Automatic variational inference with cascading flows

arXiv: https://arxiv.org/abs/2102.04801 · HTML: https://mlsanity.com/paper/2102.04801/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 41 references that are in the map.
- First posted on arXiv 9 Feb 2021.

## Builds on (most cited first, 22 of 22)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019)
- [NICE: Non-linear Independent Components Estimation](https://mlsanity.com/paper/1410.8516/) (2014)
- [Improving Variational Inference with Inverse Autoregressive Flow](https://mlsanity.com/paper/1606.04934/) (2016)
- [Augmented Neural ODEs](https://mlsanity.com/paper/1904.01681/) (2019)
- [Pyro: Deep Universal Probabilistic Programming](https://mlsanity.com/paper/1810.09538/) (2018)
- [Highway Networks](https://mlsanity.com/paper/1505.00387/) (2015)
- [Residual Flows for Invertible Generative Modeling](https://mlsanity.com/paper/1906.02735/) (2019)
- [Automatic Differentiation Variational Inference](https://mlsanity.com/paper/1603.00788/) (2016)
- [Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data](https://mlsanity.com/paper/1605.06432/) (2016)
- [Hierarchical Variational Models](https://mlsanity.com/paper/1511.02386/) (2015)
- [Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models](https://mlsanity.com/paper/2002.07101/) (2020)
- [Bayesian Recurrent Neural Networks](https://mlsanity.com/paper/1704.02798/) (2017)
- [Automated Variational Inference in Probabilistic Programming](https://mlsanity.com/paper/1301.1299/) (2013)
- [Edward: A library for probabilistic modeling, inference, and criticism](https://mlsanity.com/paper/1610.09787/) (2016)
- [Deep Probabilistic Programming](https://mlsanity.com/paper/1701.03757/) (2017)
- [Structured Stochastic Variational Inference](https://mlsanity.com/paper/1404.4114/) (2014)
- [Automatic structured variational inference](https://mlsanity.com/paper/2002.00643/) (2020)
- [Variational Inference with Continuously-Indexed Normalizing Flows](https://mlsanity.com/paper/2007.05426/) (2020)
- [Stochastic Variational Inference for Hidden Markov Models](https://mlsanity.com/paper/1411.1670/) (2014)

## Built on by (most cited first, 9 of 9)

- [ADAVI: Automatic Dual Amortized Variational Inference Applied To Pyramidal Bayesian Models](https://mlsanity.com/paper/2106.12248/) (2021)
- [Provably Scalable Black-Box Variational Inference with Structured Variational Families](https://mlsanity.com/paper/2401.10989/) (2024)
- [Embedded-model flows: Combining the inductive biases of model-free deep learning and explicit probabilistic modeling](https://mlsanity.com/paper/2110.06021/) (2021)
- [Model-Informed Flows for Bayesian Inference](https://mlsanity.com/paper/2505.24243/) (2025)
- [PAVI: Plate-Amortized Variational Inference](https://mlsanity.com/paper/2206.05111/) (2022)
- [Meta-Learning an Inference Algorithm for Probabilistic Programs](https://mlsanity.com/paper/2103.00737/) (2021)
- [PAVI: Plate-Amortized Variational Inference](https://mlsanity.com/paper/2308.16022/) (2023)
- [Understanding and mitigating difficulties in posterior predictive evaluation](https://mlsanity.com/paper/2405.19747/) (2024)
- [Causal Posterior Estimation](https://mlsanity.com/paper/2505.21468/) (2025)

## Abstract (arXiv)

The automation of probabilistic reasoning is one of the primary aims of machine learning. Recently, the confluence of variational inference and deep learning has led to powerful and flexible automatic inference methods that can be trained by stochastic gradient descent. In particular, normalizing flows are highly parameterized deep models that can fit arbitrarily complex posterior densities. However, normalizing flows struggle in highly structured probabilistic programs as they need to relearn the forward-pass of the program. Automatic structured variational inference (ASVI) remedies this problem by constructing variational programs that embed the forward-pass. Here, we combine the flexibility of normalizing flows and the prior-embedding property of ASVI in a new family of variational programs, which we named cascading flows. A cascading flows program interposes a newly designed highway flow architecture in between the conditional distributions of the prior program such as to steer it toward the observed data. These programs can be constructed automatically from an input probabilistic program and can also be amortized automatically. We evaluate the performance of the new variational programs in a series of structured inference problems. We find that cascading flows have much higher performance than both normalizing flows and ASVI in a large set of structured inference problems.