# Amortized Variational Inference for Simple Hierarchical Models

arXiv: https://arxiv.org/abs/2111.03144 · HTML: https://mlsanity.com/paper/2111.03144/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 40 references that are in the map.
- First posted on arXiv 4 Nov 2021.

## Builds on (most cited first, 15 of 15)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019)
- [Black Box Variational Inference](https://mlsanity.com/paper/1401.0118/) (2013)
- [Automatic Differentiation Variational Inference](https://mlsanity.com/paper/1603.00788/) (2016)
- [Towards a Neural Statistician](https://mlsanity.com/paper/1606.02185/) (2016)
- [DIVA: Domain Invariant Variational Autoencoders](https://mlsanity.com/paper/1905.10427/) (2019)
- [Composing graphical models with neural networks for structured representations and fast inference](https://mlsanity.com/paper/1603.06277/) (2016)
- [Multi-Level Variational Autoencoder: Learning Disentangled Representations from Grouped Observations](https://mlsanity.com/paper/1705.08841/) (2017)
- [Sticking the Landing: Simple, Lower-Variance Gradient Estimators for Variational Inference](https://mlsanity.com/paper/1703.09194/) (2017)
- [Advances in Black-Box VI: Normalizing Flows, Importance Weighting, and Optimization](https://mlsanity.com/paper/2006.10343/) (2020)
- [Provable Smoothness Guarantees for Black-Box Variational Inference](https://mlsanity.com/paper/1901.08431/) (2019)
- [Automatic structured variational inference](https://mlsanity.com/paper/2002.00643/) (2020)
- [Monte Carlo Structured SVI for Two-Level Non-Conjugate Models](https://mlsanity.com/paper/1612.03957/) (2016)
- [What are the most important statistical ideas of the past 50 years?](https://mlsanity.com/paper/2012.00174/) (2020)

## Built on by (most cited first, 19 of 19)

- [Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Training](https://mlsanity.com/paper/2602.01511/) (2026)
- [Improved off-policy training of diffusion samplers](https://mlsanity.com/paper/2402.05098/) (2024)
- [GFlowNet-EM for learning compositional latent variable models](https://mlsanity.com/paper/2302.06576/) (2023)
- [Amortized Variational Inference: When and Why?](https://mlsanity.com/paper/2307.11018/) (2023)
- [On the Convergence of Black-Box Variational Inference](https://mlsanity.com/paper/2305.15349/) (2023)
- [Amortized Bayesian Multilevel Models](https://mlsanity.com/paper/2408.13230/) (2024)
- [The Shrinkage-Delinkage Trade-off: An Analysis of Factorized Gaussian Approximations for Variational Inference](https://mlsanity.com/paper/2302.09163/) (2023)
- [Hierarchical Neural Simulation-Based Inference Over Event Ensembles](https://mlsanity.com/paper/2306.12584/) (2023)
- [Variational Inference with Locally Enhanced Bounds for Hierarchical Models](https://mlsanity.com/paper/2203.04432/) (2022)
- [Provably Scalable Black-Box Variational Inference with Structured Variational Families](https://mlsanity.com/paper/2401.10989/) (2024)
- [Fully Bayesian Differential Gaussian Processes through Stochastic Differential Equations](https://mlsanity.com/paper/2408.06069/) (2024)
- [Diffusion Bridge Variational Inference for Deep Gaussian Processes](https://mlsanity.com/paper/2509.19078/) (2025)
- [PAVI: Plate-Amortized Variational Inference](https://mlsanity.com/paper/2206.05111/) (2022)
- [Gaussian Process-Gated Hierarchical Mixtures of Experts](https://mlsanity.com/paper/2302.04947/) (2023)
- [Amortized Variational Inference for Deep Gaussian Processes](https://mlsanity.com/paper/2409.12301/) (2024)
- [PAVI: Plate-Amortized Variational Inference](https://mlsanity.com/paper/2308.16022/) (2023)
- [Understanding and mitigating difficulties in posterior predictive evaluation](https://mlsanity.com/paper/2405.19747/) (2024)
- [VariBASed: Variational Bayes-Adaptive Sequential Monte-Carlo Planning for Deep Reinforcement Learning](https://mlsanity.com/paper/2602.18857/) (2026)
- [Massively Parallel Expectation Maximization For Approximate Posteriors](https://mlsanity.com/paper/2503.08264/) (2025)

## Abstract (arXiv)

It is difficult to use subsampling with variational inference in hierarchical models since the number of local latent variables scales with the dataset. Thus, inference in hierarchical models remains a challenge at large scale. It is helpful to use a variational family with structure matching the posterior, but optimization is still slow due to the huge number of local distributions. Instead, this paper suggests an amortized approach where shared parameters simultaneously represent all local distributions. This approach is similarly accurate as using a given joint distribution (e.g., a full-rank Gaussian) but is feasible on datasets that are several orders of magnitude larger. It is also dramatically faster than using a structured variational distribution.