# Disentangling impact of capacity, objective, batchsize, estimators, and step-size on flow VI

arXiv: https://arxiv.org/abs/2412.08824 · HTML: https://mlsanity.com/paper/2412.08824/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 101 references that are in the map.
- First posted on arXiv 11 Dec 2024.

## Builds on (most cited first, 30 of 44)

- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019)
- [NICE: Non-linear Independent Components Estimation](https://mlsanity.com/paper/1410.8516/) (2014)
- [Importance Weighted Autoencoders](https://mlsanity.com/paper/1509.00519/) (2015)
- [Black Box Variational Inference](https://mlsanity.com/paper/1401.0118/) (2013)
- [A Conceptual Introduction to Hamiltonian Monte Carlo](https://mlsanity.com/paper/1701.02434/) (2017)
- [Composable Effects for Flexible and Accelerated Probabilistic Programming in NumPyro](https://mlsanity.com/paper/1912.11554/) (2019)
- [Optimal Transport Tools (OTT): A JAX Toolbox for all things Wasserstein](https://mlsanity.com/paper/2201.12324/) (2022)
- [VarGrad: A Low-Variance Gradient Estimator for Variational Inference](https://mlsanity.com/paper/2010.10436/) (2020)
- [NeuTra-lizing Bad Geometry in Hamiltonian Monte Carlo Using Neural Transport](https://mlsanity.com/paper/1903.03704/) (2019)
- [Understanding and Mitigating Exploding Inverses in Invertible Neural Networks](https://mlsanity.com/paper/2006.09347/) (2020)
- [Continual Repeated Annealed Flow Transport Monte Carlo](https://mlsanity.com/paper/2201.13117/) (2022)
- [Beyond ELBOs: A Large-Scale Evaluation of Variational Methods for Sampling](https://mlsanity.com/paper/2406.07423/) (2024)
- [Variational methods for simulation-based inference](https://mlsanity.com/paper/2203.04176/) (2022)
- [Advances in Black-Box VI: Normalizing Flows, Importance Weighting, and Optimization](https://mlsanity.com/paper/2006.10343/) (2020)
- [Local-Global MCMC kernels: the best of both worlds](https://mlsanity.com/paper/2111.02702/) (2021)
- [Batch and match: black-box variational inference with a score-based divergence](https://mlsanity.com/paper/2402.14758/) (2024)
- [Robust, Accurate Stochastic Optimization for Variational Inference](https://mlsanity.com/paper/2009.00666/) (2020)
- [On the Difficulty of Unbiased Alpha Divergence Minimization](https://mlsanity.com/paper/2010.09541/) (2020)
- [Multilevel Monte Carlo Variational Inference](https://mlsanity.com/paper/1902.00468/) (2019)
- [Generalized Doubly Reparameterized Gradient Estimators](https://mlsanity.com/paper/2101.11046/) (2021)
- [Transport Elliptical Slice Sampling](https://mlsanity.com/paper/2210.10644/) (2022)
- [MixFlows: principled variational inference via mixed flows](https://mlsanity.com/paper/2205.07475/) (2022)
- [Variational Inference for Uncertainty Quantification: an Analysis of Trade-offs](https://mlsanity.com/paper/2403.13748/) (2024)
- [tfp.mcmc: Modern Markov Chain Monte Carlo Tools Built for Modern Hardware](https://mlsanity.com/paper/2002.01184/) (2020)
- [Joint Stochastic Approximation and Its Application to Learning Discrete Latent Variable Models](https://mlsanity.com/paper/2005.14001/) (2020)
- [Empirical Evaluation of Biased Methods for Alpha Divergence Minimization](https://mlsanity.com/paper/2105.06587/) (2021)
- [Nested hat R: Assessing the convergence of Markov chain Monte Carlo when running many short chains](https://mlsanity.com/paper/2110.13017/) (2021)
- [Stochastic Variational Inference via Upper Bound](https://mlsanity.com/paper/1912.00650/) (2019)
- [Bridging the Gap Between Variational Inference and Wasserstein Gradient Flows](https://mlsanity.com/paper/2310.20090/) (2023)

## Built on by (most cited first, 5 of 5)

- [Path Gradients after Flow Matching](https://mlsanity.com/paper/2505.10139/) (2025)
- [Model-Informed Flows for Bayesian Inference](https://mlsanity.com/paper/2505.24243/) (2025)
- [Empirical evaluation of normalizing flows in Markov Chain Monte Carlo](https://mlsanity.com/paper/2412.17136/) (2024)
- [Rotated Mean-Field Variational Inference and Iterative Gaussianization](https://mlsanity.com/paper/2510.07732/) (2025)
- [Amortized Factor Inference Networks for Posterior Inference](https://mlsanity.com/paper/2605.26419/) (2026)

## Abstract (arXiv)

Normalizing flow-based variational inference (flow VI) is a promising approximate inference approach, but its performance remains inconsistent across studies. Numerous algorithmic choices influence flow VI's performance. We conduct a step-by-step analysis to disentangle the impact of some of the key factors: capacity, objectives, gradient estimators, number of gradient estimates (batchsize), and step-sizes. Each step examines one factor while neutralizing others using insights from the previous steps and/or using extensive parallel computation. To facilitate high-fidelity evaluation, we curate a benchmark of synthetic targets that represent common posterior pathologies and allow for exact sampling. We provide specific recommendations for different factors and propose a flow VI recipe that matches or surpasses leading turnkey Hamiltonian Monte Carlo (HMC) methods.