# Automatic Differentiation Variational Inference with Mixtures

arXiv: https://arxiv.org/abs/2003.01687 · HTML: https://mlsanity.com/paper/2003.01687/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Mar 2020.

## Built on by (most cited first, 19 of 19)

- [Mitigating Modality Collapse in Multimodal VAEs via Impartial Optimization](https://mlsanity.com/paper/2206.04496/) (2022)
- [A Unified Perspective on Natural Gradient Variational Inference with Gaussian Mixture Models](https://mlsanity.com/paper/2209.11533/) (2022)
- [VaiPhy: a Variational Inference Based Algorithm for Phylogeny](https://mlsanity.com/paper/2203.01121/) (2022)
- [Particle Semi-Implicit Variational Inference](https://mlsanity.com/paper/2407.00649/) (2024)
- [Cooperation in the Latent Space: The Benefits of Adding Mixture Components in Variational Autoencoders](https://mlsanity.com/paper/2209.15514/) (2022)
- [Learning multi-modal generative models with permutation-invariant encoders and tighter variational objectives](https://mlsanity.com/paper/2309.00380/) (2023)
- [Improved Variational Bayesian Phylogenetic Inference using Mixtures](https://mlsanity.com/paper/2310.00941/) (2023)
- [Nonparametric Automatic Differentiation Variational Inference with Spline Approximation](https://mlsanity.com/paper/2403.06302/) (2024)
- [Efficient Mixture Learning in Black-Box Variational Inference](https://mlsanity.com/paper/2406.07083/) (2024)
- [Sparse Kernel Gaussian Processes through Iterative Charted Refinement (ICR)](https://mlsanity.com/paper/2206.10634/) (2022)
- [Improving Variational Autoencoder Estimation from Incomplete Data with Mixture Variational Families](https://mlsanity.com/paper/2403.03069/) (2024)
- [Fisher meets Feynman: score-based variational inference with a product of experts](https://mlsanity.com/paper/2510.21598/) (2025)
- [A Kernel Approach for Semi-implicit Variational Inference](https://mlsanity.com/paper/2601.12023/) (2026)
- [How to Approximate Inference with Subtractive Mixture Models](https://mlsanity.com/paper/2604.16714/) (2026)
- [Even More Guarantees for Variational Inference in the Presence of Symmetries](https://mlsanity.com/paper/2604.21407/) (2026)
- [Beyond Bayesian Model Averaging over Paths in Probabilistic Programs with Stochastic Support](https://mlsanity.com/paper/2310.14888/) (2023)
- [Quasi Monte Carlo methods enable extremely low-dimensional deep generative models](https://mlsanity.com/paper/2601.18676/) (2026)
- [Learning Mixture Density via Natural Gradient Expectation Maximization](https://mlsanity.com/paper/2602.10602/) (2026)
- [Structured Nonparametric Variational Inference for Dependent Latent Modeling](https://mlsanity.com/paper/2606.15458/) (2026)

## Abstract (arXiv)

Automatic Differentiation Variational Inference (ADVI) is a useful tool for efficiently learning probabilistic models in machine learning. Generally approximate posteriors learned by ADVI are forced to be unimodal in order to facilitate use of the reparameterization trick. In this paper, we show how stratified sampling may be used to enable mixture distributions as the approximate posterior, and derive a new lower bound on the evidence analogous to the importance weighted autoencoder (IWAE). We show that this "SIWAE" is a tighter bound than both IWAE and the traditional ELBO, both of which are special instances of this bound. We verify empirically that the traditional ELBO objective disfavors the presence of multimodal posterior distributions and may therefore not be able to fully capture structure in the latent space. Our experiments show that using the SIWAE objective allows the encoder to learn more complex distributions which regularly contain multimodality, resulting in higher accuracy and better calibration in the presence of incomplete, limited, or corrupted data.