# FlowVAT: Normalizing Flow Variational Inference with Affine-Invariant Tempering

arXiv: https://arxiv.org/abs/2505.10466 · HTML: https://mlsanity.com/paper/2505.10466/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 41 references that are in the map.
- First posted on arXiv 15 May 2025.

## Builds on (most cited first, 27 of 27)

- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/1702.03118/) (2017)
- [Neural Spline Flows](https://mlsanity.com/paper/1906.04032/) (2019)
- [Importance Weighted Autoencoders](https://mlsanity.com/paper/1509.00519/) (2015)
- [Pyro: Deep Universal Probabilistic Programming](https://mlsanity.com/paper/1810.09538/) (2018)
- [Black Box Variational Inference](https://mlsanity.com/paper/1401.0118/) (2013)
- [Composable Effects for Flexible and Accelerated Probabilistic Programming in NumPyro](https://mlsanity.com/paper/1912.11554/) (2019)
- [R'enyi Divergence Variational Inference](https://mlsanity.com/paper/1602.02311/) (2016)
- [NeuTra-lizing Bad Geometry in Hamiltonian Monte Carlo Using Neural Transport](https://mlsanity.com/paper/1903.03704/) (2019)
- [Pareto Smoothed Importance Sampling](https://mlsanity.com/paper/1507.02646/) (2015)
- [Advances in Black-Box VI: Normalizing Flows, Importance Weighting, and Optimization](https://mlsanity.com/paper/2006.10343/) (2020)
- [Double Descent Demystified: Identifying, Interpreting & Ablating the Sources of a Deep Learning Puzzle](https://mlsanity.com/paper/2303.14151/) (2023)
- [Challenges and Opportunities in High-dimensional Variational Inference](https://mlsanity.com/paper/2103.01085/) (2021)
- [Marginal likelihood computation for model selection and hypothesis testing: an extensive review](https://mlsanity.com/paper/2005.08334/) (2020)
- [Variational Inference with Tail-adaptive f-Divergence](https://mlsanity.com/paper/1810.11943/) (2018)
- [Real-time gravitational-wave inference for binary neutron stars using machine learning](https://mlsanity.com/paper/2407.09602/) (2024)
- [Variational Tempering](https://mlsanity.com/paper/1411.1810/) (2014)
- [Black Box Variational Inference with a Deterministic Objective: Faster, More Accurate, and Even More Black Box](https://mlsanity.com/paper/2304.05527/) (2023)
- [A theoretical perspective on mode collapse in variational inference](https://mlsanity.com/paper/2410.13300/) (2024)
- [Alpha-Beta Divergence For Variational Inference](https://mlsanity.com/paper/1805.01045/) (2018)
- [Annealing Flow Generative Models Towards Sampling High-Dimensional and Multi-Modal Distributions](https://mlsanity.com/paper/2409.20547/) (2024)
- [Learned harmonic mean estimation of the Bayesian evidence with normalizing flows](https://mlsanity.com/paper/2405.05969/) (2024)
- [NAUTILUS: boosting Bayesian importance nested sampling with deep learning](https://mlsanity.com/paper/2306.16923/) (2023)
- [Bayesian evidence estimation from posterior samples with normalizing flows](https://mlsanity.com/paper/2404.12294/) (2024)
- [AdaAnn: Adaptive Annealing Scheduler for Probability Density Approximation](https://mlsanity.com/paper/2202.00792/) (2022)
- [Fast Bayesian Inference for Neutrino Non-Standard Interactions at Dark Matter Direct Detection Experiments](https://mlsanity.com/paper/2405.14932/) (2024)

## Built on by (most cited first, 2 of 2)

- [Generative Nested Sampling of Atomistic Thermodynamic Landscapes](https://mlsanity.com/paper/2609.03193/) (2026)
- [Mode Collapse Is Cheap to Detect: A Ground-Truth-Free Pre-Flight Check for Neural Samplers](https://mlsanity.com/paper/2609.26272/) (2026)

## Abstract (arXiv)

Multi-modal and high-dimensional posteriors present significant challenges for variational inference, causing mode-seeking behavior and collapse despite the theoretical expressiveness of normalizing flows. Traditional annealing methods require temperature schedules and hyperparameter tuning, falling short of the goal of truly black-box variational inference. We introduce FlowVAT, a conditional tempering approach for normalizing flow variational inference that addresses these limitations. Our method tempers both the base and target distributions simultaneously, maintaining affine-invariance under tempering. By conditioning the normalizing flow on temperature, we leverage overparameterized neural networks' generalization capabilities to train a single flow representing the posterior across a range of temperatures. This preserves modes identified at higher temperatures when sampling from the variational posterior at T = 1, mitigating standard variational methods' mode-seeking behavior. In experiments with 2, 10, and 20 dimensional multi-modal distributions, FlowVAT outperforms traditional and adaptive annealing methods, finding more modes and achieving better ELBO values, particularly in higher dimensions where existing approaches fail. Our method requires minimal hyperparameter tuning and does not require an annealing schedule, advancing toward fully-automatic black-box variational inference for complicated posteriors.