mlsanityData to 7 Oct 2026

U-Statistics for Importance-Weighted Variational Inference

arXiv 2302.13918 ↗First posted 27 Feb 2023cs.LG stat.MLcited by 3

Not on the map. Why?

Abstract (arXiv)

We propose the use of U-statistics to reduce variance for gradient estimation in importance-weighted variational inference. The key observation is that, given a base gradient estimator that requires m > 1 samples and a total of n > m samples to be used for estimation, lower variance is achieved by averaging the base estimator on overlapping batches of size m than disjoint batches, as currently done. We use classical U-statistic theory to analyze the variance reduction, and propose novel approximations with theoretical guarantees to ensure computational efficiency. We find empirically that U-statistic variance reduction can lead to modest to significant improvements in inference performance on a range of models, with little computational cost.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 30 of its 54 references that are in the map. First posted on arXiv 27 Feb 2023.

Builds on

30 of 54 references.

References · 30

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  4. 2013Auto-Encoding Variational Bayescited by 3,980
  5. 2016Variational Inference: A Review for Statisticianscited by 1,466
  6. 2015Importance Weighted Autoencoderscited by 454
  7. 2018Deep Learning for Classical Japanese Literaturecited by 415
  8. 2018Pyro: Deep Universal Probabilistic Programmingcited by 379
  9. 2013Black Box Variational Inferencecited by 317
  10. 2020Fast Differentiable Sorting and Rankingcited by 196
  11. 2014Neural Variational Inference and Learning in Belief Networkscited by 116
  12. 2017Variational Sequential Monte Carlocited by 93
18 more
  1. 2017Filtering Variational Objectivescited by 91
  2. 2016Variational inference for Monte Carlo objectivescited by 83
  3. 2018Tighter Variational Bounds are Not Necessarily Bettercited by 76
  4. 2014Reweighted Wake-Sleepcited by 66
  5. 2017Auto-Encoding Sequential Monte Carlocited by 64
  6. 2018Importance Weighting and Variational Inferencecited by 55
  7. 2018Doubly Reparameterized Gradient Estimators for Monte Carlo Objectivescited by 41
  8. 2020Advances in Black-Box VI: Normalizing Flows, Importance Weighting, and Optimizationcited by 37
  9. 2017Reinterpreting Importance-Weighted Autoencoderscited by 36
  10. 2017Reducing Reparameterization Gradient Variancecited by 27
  11. 2018Variational Inference with Tail-adaptive f-Divergencecited by 25
  12. 2018Using Large Ensembles of Control Variates for Variational Inferencecited by 21
  13. 2019Divide and Couple: Using Monte Carlo Variational Objectives for Posterior Approximationcited by 14
  14. 2021Generalized Doubly Reparameterized Gradient Estimatorscited by 14
  15. 2020Approximation Based Variance Reduction for Reparameterization Gradientscited by 10
  16. 2019Reweighted Expectation Maximizationcited by 9
  17. 2020Optimal Variance Control of the Score Function Gradient Estimator for Importance Weighted Boundscited by 7
  18. 2022Uphill Roads to Variational Tightness: Monotonicity and Monte Carlo Objectivescited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 0; 2026: 1.

3 papers.

  1. 2023Sample Average Approximation for Black-Box VIcited by 6
  2. 2024Disentangling impact of capacity, objective, batchsize, estimators, and step-size on flow VIcited by 5
  3. 2026SGD for Variational Inference: Tackling Unbounded Variance via Preconditioning and Dynamic Batchingcited by 0