mlsanityData to 7 Oct 2026

Tighter Variational Bounds are Not Necessarily Better

arXiv 1802.04537 ↗First posted 13 Feb 2018stat.ML cs.LGcited by 76

Not on the map. Why?

Abstract (arXiv)

We provide theoretical and empirical evidence that using tighter evidence lower bounds (ELBOs) can be detrimental to the process of learning an inference network by reducing the signal-to-noise ratio of the gradient estimator. Our results call into question common implicit assumptions that tighter ELBOs are better variational objectives for simultaneous model learning and inference amortization schemes. Based on our insights, we introduce three new algorithms: the partially importance weighted auto-encoder (PIWAE), the multiply importance weighted auto-encoder (MIWAE), and the combination importance weighted auto-encoder (CIWAE), each of which includes the standard importance weighted auto-encoder (IWAE) as a special case. We show that each can deliver improvements over IWAE, even when performance is measured by the IWAE target itself. Furthermore, our results suggest that PIWAE may be able to deliver simultaneous improvements in the training of both the inference and generative networks.

Key facts. Cited by 76 papers in the map (data through 7 Oct 2026). First posted on arXiv 13 Feb 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 2; 2020: 0; 2021: 14; 2022: 17; 2023: 13; 2024: 12; 2025: 7; 2026: 11.

76 papers.

Sort:

By area:

  1. 2019Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Modelscited by 177
  2. 2022GFlowNets and variational inferencecited by 103
  3. 2022Variational methods for simulation-based inferencecited by 43
  4. 2022Mitigating Modality Collapse in Multimodal VAEs via Impartial Optimizationcited by 32
  5. 2021Variational Refinement for Importance Sampling Using the Forward Kullback-Leibler Divergencecited by 29
  6. 2025Beyond Verifiable Rewards: Scaling Reinforcement Learning for Language Models to Unverifiable Datacited by 29
  7. 2021Decomposed Mutual Information Estimation for Contrastive Representation Learningcited by 24
  8. 2023SeeGera: Self-supervised Semi-implicit Graph Variational Auto-encoders with Maskingcited by 23
  9. 2021Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCEcited by 22
  10. 2021Differentiable Particle Filtering without Modifying the Forward Passcited by 18
  11. 2021VACA: Design of Variational Graph Autoencoders for Interventional and Counterfactual Queriescited by 18
  12. 2022Amortized Variational Inference: A Systematic Reviewcited by 16
38 more
  1. 2022Alpha-divergence Variational Inference Meets Importance Weighted Auto-Encoders: Methodology and Asymptoticscited by 15
  2. 2021Generalized Doubly Reparameterized Gradient Estimatorscited by 14
  3. 2021Nested Variational Inferencecited by 14
  4. 2022Latent State Marginalization as a Low-cost Approach for Improving Explorationcited by 13
  5. 2023More Synergy, Less Redundancy: Exploiting Joint Mutual Information for Self-Supervised Learningcited by 11
  6. 2023Linear Convergence of Black-Box Variational Inference: Should We Stick the Landing?cited by 10
  7. 2022A Variational Approach to Bayesian Phylogenetic Inferencecited by 8
  8. 2023Delta-AI: Local objectives for amortized inference in sparse graphical modelscited by 7
  9. 2024Causal Contrastive Learning for Counterfactual Regression Over Timecited by 7
  10. 2021Learning Proposals for Probabilistic Programs with Inference Combinatorscited by 6
  11. 2021Variational Marginal Particle Filterscited by 6
  12. 2022Variational Sparse Coding with Learned Thresholdingcited by 6
  13. 2022Bounding Evidence and Estimating Log-Likelihood in VAEcited by 6
  14. 2024Variance reduction of diffusion model's gradients with Taylor approximation-based control variatecited by 6
  15. 2022Multi-Sample Training for Neural Image Compressioncited by 5
  16. 2022Variational Open-Domain Question Answeringcited by 5
  17. 2023NAS-X: Neural Adaptive Smoothing via Twistingcited by 4
  18. 2023ARTree: A Deep Autoregressive Model for Phylogenetic Inferencecited by 4
  19. 2024Probabilistic Programming with Programmable Variational Inferencecited by 4
  20. 2025In-Context Parametric Inference: Point or Distribution Estimators?cited by 4
  21. 2021Variational Inference with Holder Boundscited by 3
  22. 2022Uphill Roads to Variational Tightness: Monotonicity and Monte Carlo Objectivescited by 3
  23. 2022Learning to Drop Out: An Adversarial Approach to Training Sequence VAEscited by 3
  24. 2023U-Statistics for Importance-Weighted Variational Inferencecited by 3
  25. 2023Accelerating Large Batch Training via Gradient Signal to Noise Ratio (GSNR)cited by 3
  26. 2024Nonparametric Automatic Differentiation Variational Inference with Spline Approximationcited by 3
  27. 2024Efficient Mixture Learning in Black-Box Variational Inferencecited by 3
  28. 2024Variational Learning of Gaussian Process Latent Variable Models through Stochastic Gradient Annealed Importance Samplingcited by 3
  29. 2022Reconstruction of Incomplete Wildfire Data using Deep Generative Modelscited by 2
  30. 2023Unscented Autoencodercited by 2
  31. 2024Improving Variational Autoencoder Estimation from Incomplete Data with Mixture Variational Familiescited by 2
  32. 2025DeepKD: A Deeply Decoupled and Denoised Knowledge Distillation Trainercited by 2
  33. 2026On the Difficulty of Learning a Meta-network for Training Data Selectioncited by 2
  34. 2021Monte Carlo Filtering Objectives: A New Family of Variational Objectives to Learn Generative Model and Neural Adaptive Proposal for Time Seriescited by 1
  35. 2022Structured Recognition for Generative Models with Explaining Awaycited by 1
  36. 2022Fast and Efficient Scene Categorization for Autonomous Driving using VAEscited by 1
  37. 2023The Deep Latent Position Topic Model for Clustering and Representation of Networks with Textual Edgescited by 1
  38. 2023On the Natural Gradient of the Evidence Lower Boundcited by 1

50 most cited of 76 All 76 (JSON)