# ADAVI: Automatic Dual Amortized Variational Inference Applied To Pyramidal Bayesian Models

arXiv: https://arxiv.org/abs/2106.12248 · HTML: https://mlsanity.com/paper/2106.12248/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 66 references that are in the map.
- First posted on arXiv 23 Jun 2021.

## Builds on (most cited first, 30 of 31)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019)
- [Geometric deep learning: going beyond Euclidean data](https://mlsanity.com/paper/1611.08097/) (2016)
- [Masked Autoregressive Flow for Density Estimation](https://mlsanity.com/paper/1705.07057/) (2017)
- [FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models](https://mlsanity.com/paper/1810.01367/) (2018)
- [The frontier of simulation-based inference](https://mlsanity.com/paper/1911.01429/) (2019)
- [Pyro: Deep Universal Probabilistic Programming](https://mlsanity.com/paper/1810.09538/) (2018)
- [Black Box Variational Inference](https://mlsanity.com/paper/1401.0118/) (2013)
- [Advances in Variational Inference](https://mlsanity.com/paper/1711.05597/) (2017)
- [Sequential Neural Likelihood: Fast Likelihood-free Inference with Autoregressive Flows](https://mlsanity.com/paper/1805.07226/) (2018)
- [Automatic Posterior Transformation for Likelihood-Free Inference](https://mlsanity.com/paper/1905.07488/) (2019)
- [Automatic Differentiation Variational Inference](https://mlsanity.com/paper/1603.00788/) (2016)
- [BayesFlow: Learning complex stochastic models with invertible neural networks](https://mlsanity.com/paper/2003.06281/) (2020)
- [Inference Suboptimality in Variational Autoencoders](https://mlsanity.com/paper/1801.03558/) (2018)
- [TensorFlow Distributions](https://mlsanity.com/paper/1711.10604/) (2017)
- [Hierarchical Variational Models](https://mlsanity.com/paper/1511.02386/) (2015)
- [Likelihood-free inference by ratio estimation](https://mlsanity.com/paper/1611.10242/) (2016)
- [Fast ε-free Inference of Simulation Models with Bayesian Conditional Density Estimation](https://mlsanity.com/paper/1605.06376/) (2016)
- [Graphical Normalizing Flows](https://mlsanity.com/paper/2006.02548/) (2020)
- [Amortized Inference Regularization](https://mlsanity.com/paper/1805.08913/) (2018)
- [Faithful Inversion of Generative Models for Effective Amortized Inference](https://mlsanity.com/paper/1712.00287/) (2017)
- [Structured Stochastic Variational Inference](https://mlsanity.com/paper/1404.4114/) (2014)
- [Meta-Amortized Variational Inference and Learning](https://mlsanity.com/paper/1902.01950/) (2019)
- [Automatic structured variational inference](https://mlsanity.com/paper/2002.00643/) (2020)
- [Meta-Learning with Shared Amortized Variational Inference](https://mlsanity.com/paper/2008.12037/) (2020)
- [Automatic variational inference with cascading flows](https://mlsanity.com/paper/2102.04801/) (2021)

## Built on by (most cited first, 4 of 4)

- [Validation Diagnostics for SBI algorithms based on Normalizing Flows](https://mlsanity.com/paper/2211.09602/) (2022)
- [PAVI: Plate-Amortized Variational Inference](https://mlsanity.com/paper/2206.05111/) (2022)
- [PAVI: Plate-Amortized Variational Inference](https://mlsanity.com/paper/2308.16022/) (2023)
- [Semi-Amortized Bayesian Inference with Reusable Local Posteriors](https://mlsanity.com/paper/2610.18945/) (2026)

## Abstract (arXiv)

Frequently, population studies feature pyramidally-organized data represented using Hierarchical Bayesian Models (HBM) enriched with plates. These models can become prohibitively large in settings such as neuroimaging, where a sample is composed of a functional MRI signal measured on 300 brain locations, across 4 measurement sessions, and 30 subjects, resulting in around 1 million latent parameters.Such high dimensionality hampers the usage of modern, expressive flow-based techniques.To infer parameter posterior distributions in this challenging class of problems, we designed a novel methodology that automatically produces a variational family dual to a target HBM. This variational family, represented as a neural network, consists in the combination of an attention-based hierarchical encoder feeding summary statistics to a set of normalizing flows. Our automatically-derived neural network exploits exchangeability in the plate-enriched HBM and factorizes its parameter space. The resulting architecture reduces by orders of magnitude its parameterization with respect to that of a typical flow-based representation, while maintaining expressivity.Our method performs inference on the specified HBM in an amortized setup: once trained, it can readily be applied to a new data sample to compute the parameters' full posterior.We demonstrate the capability and scalability of our method on simulated data, as well as a challenging high-dimensional brain parcellation experiment. We also open up several questions that lie at the intersection between normalizing flows, SBI, structured Variational Inference, and inference amortization.