# Advances in Black-Box VI: Normalizing Flows, Importance Weighting, and Optimization

arXiv: https://arxiv.org/abs/2006.10343 · HTML: https://mlsanity.com/paper/2006.10343/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jun 2020.

## Built on by (most cited first, 30 of 37)

- [Variational methods for simulation-based inference](https://mlsanity.com/paper/2203.04176/) (2022)
- [Challenges and Opportunities in High-dimensional Variational Inference](https://mlsanity.com/paper/2103.01085/) (2021)
- [Amortized Variational Inference: When and Why?](https://mlsanity.com/paper/2307.11018/) (2023)
- [Gradients should stay on Path: Better Estimators of the Reverse- and Forward KL Divergence for Normalizing Flows](https://mlsanity.com/paper/2207.08219/) (2022)
- [On the Convergence of Black-Box Variational Inference](https://mlsanity.com/paper/2305.15349/) (2023)
- [Amortized Variational Inference for Simple Hierarchical Models](https://mlsanity.com/paper/2111.03144/) (2021)
- [Variational Inference with Gaussian Score Matching](https://mlsanity.com/paper/2307.07849/) (2023)
- [Generalized Doubly Reparameterized Gradient Estimators](https://mlsanity.com/paper/2101.11046/) (2021)
- [Fat-Tailed Variational Inference with Anisotropic Tail Adaptive Flows](https://mlsanity.com/paper/2205.07918/) (2022)
- [Path-Gradient Estimators for Continuous Normalizing Flows](https://mlsanity.com/paper/2206.09016/) (2022)
- [EigenVI: score-based variational inference with orthogonal function expansions](https://mlsanity.com/paper/2410.24054/) (2024)
- [Linear Convergence of Black-Box Variational Inference: Should We Stick the Landing?](https://mlsanity.com/paper/2307.14642/) (2023)
- [Variational Inference in Location-Scale Families: Exact Recovery of the Mean and Correlation Matrix](https://mlsanity.com/paper/2410.11067/) (2024)
- [Bridging the Gap Between Variational Inference and Wasserstein Gradient Flows](https://mlsanity.com/paper/2310.20090/) (2023)
- [Sample Average Approximation for Black-Box VI](https://mlsanity.com/paper/2304.06803/) (2023)
- [Variational Inference with Coverage Guarantees in Simulation-Based Inference](https://mlsanity.com/paper/2305.14275/) (2023)
- [Fast and Unified Path Gradient Estimators for Normalizing Flows](https://mlsanity.com/paper/2403.15881/) (2024)
- [Disentangling impact of capacity, objective, batchsize, estimators, and step-size on flow VI](https://mlsanity.com/paper/2412.08824/) (2024)
- [Provably Scalable Black-Box Variational Inference with Structured Variational Families](https://mlsanity.com/paper/2401.10989/) (2024)
- [Bernstein Flows for Flexible Posteriors in Variational Bayes](https://mlsanity.com/paper/2202.05650/) (2022)
- [U-Statistics for Importance-Weighted Variational Inference](https://mlsanity.com/paper/2302.13918/) (2023)
- [TRADE: Transfer of Distributions between External Conditions with Normalizing Flows](https://mlsanity.com/paper/2410.19492/) (2024)
- [Path Gradients after Flow Matching](https://mlsanity.com/paper/2505.10139/) (2025)
- [Model-Informed Flows for Bayesian Inference](https://mlsanity.com/paper/2505.24243/) (2025)
- [Generalized Guarantees for Variational Inference in the Presence of Even and Elliptical Symmetry](https://mlsanity.com/paper/2511.01064/) (2025)
- [Scalable Semi-Modular Inference with Variational Meta-Posteriors](https://mlsanity.com/paper/2204.00296/) (2022)
- [Discriminative calibration: Check Bayesian computation from simulations and flexible classifier](https://mlsanity.com/paper/2305.14593/) (2023)
- [Rethinking Variational Inference for Probabilistic Programs with Stochastic Support](https://mlsanity.com/paper/2311.00594/) (2023)
- [Energy-Based Coarse-Graining in Molecular Dynamics: A Flow-Based Framework without Data](https://mlsanity.com/paper/2504.20940/) (2025)
- [FlowVAT: Normalizing Flow Variational Inference with Affine-Invariant Tempering](https://mlsanity.com/paper/2505.10466/) (2025)

## Abstract (arXiv)

Recent research has seen several advances relevant to black-box VI, but the current state of automatic posterior inference is unclear. One such advance is the use of normalizing flows to define flexible posterior densities for deep latent variable models. Another direction is the integration of Monte-Carlo methods to serve two purposes; first, to obtain tighter variational objectives for optimization, and second, to define enriched variational families through sampling. However, both flows and variational Monte-Carlo methods remain relatively unexplored for black-box VI. Moreover, on a pragmatic front, there are several optimization considerations like step-size scheme, parameter initialization, and choice of gradient estimators, for which there are no clear guidance in the existing literature. In this paper, we postulate that black-box VI is best addressed through a careful combination of numerous algorithmic components. We evaluate components relating to optimization, flows, and Monte-Carlo methods on a benchmark of 30 models from the Stan model library. The combination of these algorithmic components significantly advances the state-of-the-art "out of the box" variational inference.