# U-Statistics for Importance-Weighted Variational Inference

arXiv: https://arxiv.org/abs/2302.13918 · HTML: https://mlsanity.com/paper/2302.13918/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 54 references that are in the map.
- First posted on arXiv 27 Feb 2023.

## Builds on (most cited first, 30 of 30)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Importance Weighted Autoencoders](https://mlsanity.com/paper/1509.00519/) (2015)
- [Deep Learning for Classical Japanese Literature](https://mlsanity.com/paper/1812.01718/) (2018)
- [Pyro: Deep Universal Probabilistic Programming](https://mlsanity.com/paper/1810.09538/) (2018)
- [Black Box Variational Inference](https://mlsanity.com/paper/1401.0118/) (2013)
- [Fast Differentiable Sorting and Ranking](https://mlsanity.com/paper/2002.08871/) (2020)
- [Neural Variational Inference and Learning in Belief Networks](https://mlsanity.com/paper/1402.0030/) (2014)
- [Variational Sequential Monte Carlo](https://mlsanity.com/paper/1705.11140/) (2017)
- [Filtering Variational Objectives](https://mlsanity.com/paper/1705.09279/) (2017)
- [Variational inference for Monte Carlo objectives](https://mlsanity.com/paper/1602.06725/) (2016)
- [Tighter Variational Bounds are Not Necessarily Better](https://mlsanity.com/paper/1802.04537/) (2018)
- [Reweighted Wake-Sleep](https://mlsanity.com/paper/1406.2751/) (2014)
- [Auto-Encoding Sequential Monte Carlo](https://mlsanity.com/paper/1705.10306/) (2017)
- [Importance Weighting and Variational Inference](https://mlsanity.com/paper/1808.09034/) (2018)
- [Doubly Reparameterized Gradient Estimators for Monte Carlo Objectives](https://mlsanity.com/paper/1810.04152/) (2018)
- [Reinterpreting Importance-Weighted Autoencoders](https://mlsanity.com/paper/1704.02916/) (2017)
- [Advances in Black-Box VI: Normalizing Flows, Importance Weighting, and Optimization](https://mlsanity.com/paper/2006.10343/) (2020)
- [Reducing Reparameterization Gradient Variance](https://mlsanity.com/paper/1705.07880/) (2017)
- [Variational Inference with Tail-adaptive f-Divergence](https://mlsanity.com/paper/1810.11943/) (2018)
- [Using Large Ensembles of Control Variates for Variational Inference](https://mlsanity.com/paper/1810.12482/) (2018)
- [Divide and Couple: Using Monte Carlo Variational Objectives for Posterior Approximation](https://mlsanity.com/paper/1906.10115/) (2019)
- [Generalized Doubly Reparameterized Gradient Estimators](https://mlsanity.com/paper/2101.11046/) (2021)
- [Approximation Based Variance Reduction for Reparameterization Gradients](https://mlsanity.com/paper/2007.14634/) (2020)
- [Reweighted Expectation Maximization](https://mlsanity.com/paper/1906.05850/) (2019)
- [Optimal Variance Control of the Score Function Gradient Estimator for Importance Weighted Bounds](https://mlsanity.com/paper/2008.01998/) (2020)
- [Uphill Roads to Variational Tightness: Monotonicity and Monte Carlo Objectives](https://mlsanity.com/paper/2201.10989/) (2022)

## Built on by (most cited first, 3 of 3)

- [Sample Average Approximation for Black-Box VI](https://mlsanity.com/paper/2304.06803/) (2023)
- [Disentangling impact of capacity, objective, batchsize, estimators, and step-size on flow VI](https://mlsanity.com/paper/2412.08824/) (2024)
- [SGD for Variational Inference: Tackling Unbounded Variance via Preconditioning and Dynamic Batching](https://mlsanity.com/paper/2605.07531/) (2026)

## Abstract (arXiv)

We propose the use of U-statistics to reduce variance for gradient estimation in importance-weighted variational inference. The key observation is that, given a base gradient estimator that requires m > 1 samples and a total of n > m samples to be used for estimation, lower variance is achieved by averaging the base estimator on overlapping batches of size m than disjoint batches, as currently done. We use classical U-statistic theory to analyze the variance reduction, and propose novel approximations with theoretical guarantees to ensure computational efficiency. We find empirically that U-statistic variance reduction can lead to modest to significant improvements in inference performance on a range of models, with little computational cost.