# A Contrastive Learning Approach for Training Variational Autoencoder Priors

arXiv: https://arxiv.org/abs/2010.02917 · HTML: https://mlsanity.com/paper/2010.02917/

## Key facts

- Cited by 66 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Oct 2020.

## Built on by (most cited first, 30 of 66)

- [Tackling the Generative Learning Trilemma with Denoising Diffusion GANs](https://mlsanity.com/paper/2112.07804/) (2021)
- [PhysDiff: Physics-Guided Human Motion Diffusion Model](https://mlsanity.com/paper/2212.02500/) (2022)
- [Score-Based Generative Modeling with Critically-Damped Langevin Diffusion](https://mlsanity.com/paper/2112.07068/) (2021)
- [Geometric Latent Diffusion Models for 3D Molecule Generation](https://mlsanity.com/paper/2305.01140/) (2023)
- [Diffusion-SDF: Conditional Generative Modeling of Signed Distance Functions](https://mlsanity.com/paper/2211.13757/) (2022)
- [DiffiT: Diffusion Vision Transformers for Image Generation](https://mlsanity.com/paper/2312.02139/) (2023)
- [Refining Generative Process with Discriminator Guidance in Score-based Diffusion Models](https://mlsanity.com/paper/2211.17091/) (2022)
- [Controllable and Compositional Generation with Latent-Space Energy-Based Models](https://mlsanity.com/paper/2110.10873/) (2021)
- [Generative Multiplane Images: Making a 2D GAN 3D-Aware](https://mlsanity.com/paper/2207.10642/) (2022)
- [Generative Modeling through the Semi-dual Formulation of Unbalanced Optimal Transport](https://mlsanity.com/paper/2305.14777/) (2023)
- [PaGoDA: Progressive Growing of a One-Step Generator from a Low-Resolution Diffusion Teacher](https://mlsanity.com/paper/2405.14822/) (2024)
- [Particle algorithms for maximum likelihood training of latent variable models](https://mlsanity.com/paper/2204.12965/) (2022)
- [Scalable Wasserstein Gradient Flow for Generative Modeling through Unbalanced Optimal Transport](https://mlsanity.com/paper/2402.05443/) (2024)
- [Creativity and Machine Learning: A Survey](https://mlsanity.com/paper/2104.02726/) (2021)
- [Adaptive Multi-stage Density Ratio Estimation for Learning Latent Space Energy-based Model](https://mlsanity.com/paper/2209.08739/) (2022)
- [Guiding Energy-based Models via Contrastive Latent Variables](https://mlsanity.com/paper/2303.03023/) (2023)
- [VCT: Training Consistency Models with Variational Noise Coupling](https://mlsanity.com/paper/2502.18197/) (2025)
- [Semi-Implicit Denoising Diffusion Models (SIDDMs)](https://mlsanity.com/paper/2306.12511/) (2023)
- [Variational Autoencoding Discrete Diffusion with Enhanced Dimensional Correlations Modeling](https://mlsanity.com/paper/2505.17384/) (2025)
- [Learning Energy-Based Prior Model with Diffusion-Amortized MCMC](https://mlsanity.com/paper/2310.03218/) (2023)
- [Geometric Representation Condition Improves Equivariant Molecule Generation](https://mlsanity.com/paper/2410.03655/) (2024)
- [Personalized Federated Collaborative Filtering: A Variational AutoEncoder Approach](https://mlsanity.com/paper/2408.08931/) (2024)
- [Decoupled Graph Energy-based Model for Node Out-of-Distribution Detection on Heterophilic Graphs](https://mlsanity.com/paper/2502.17912/) (2025)
- [Preference Optimization by Estimating the Ratio of the Data Distribution](https://mlsanity.com/paper/2505.19601/) (2025)
- [Learning Joint Latent Space EBM Prior Model for Multi-layer Generator](https://mlsanity.com/paper/2306.06323/) (2023)
- [Analyzing and Improving Optimal-Transport-based Adversarial Networks](https://mlsanity.com/paper/2310.02611/) (2023)
- ["Noisier" Noise Contrastive Eestimation is (Almost) Maximum Likelihood](https://mlsanity.com/paper/2405.16730/) (2024)
- [Image Generation with a Sphere Encoder](https://mlsanity.com/paper/2602.15030/) (2026)
- [Gromov-Wasserstein Autoencoders](https://mlsanity.com/paper/2209.07007/) (2022)
- [Learning Energy-based Model via Dual-MCMC Teaching](https://mlsanity.com/paper/2312.02469/) (2023)

## Abstract (arXiv)

Variational autoencoders (VAEs) are one of the powerful likelihood-based generative models with applications in many domains. However, they struggle to generate high-quality images, especially when samples are obtained from the prior without any tempering. One explanation for VAEs' poor generative quality is the prior hole problem: the prior distribution fails to match the aggregate approximate posterior. Due to this mismatch, there exist areas in the latent space with high density under the prior that do not correspond to any encoded image. Samples from those areas are decoded to corrupted images. To tackle this issue, we propose an energy-based prior defined by the product of a base prior distribution and a reweighting factor, designed to bring the base closer to the aggregate posterior. We train the reweighting factor by noise contrastive estimation, and we generalize it to hierarchical VAEs with many latent variable groups. Our experiments confirm that the proposed noise contrastive priors improve the generative performance of state-of-the-art VAEs by a large margin on the MNIST, CIFAR-10, CelebA 64, and CelebA HQ 256 datasets. Our method is simple and can be applied to a wide variety of VAEs to improve the expressivity of their prior distribution.