# Regularizing Variational Autoencoder with Diversity and Uncertainty Awareness

arXiv: https://arxiv.org/abs/2110.12381 · HTML: https://mlsanity.com/paper/2110.12381/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 55 references that are in the map.
- First posted on arXiv 24 Oct 2021.

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [Conditional Image Generation with PixelCNN Decoders](https://mlsanity.com/paper/1606.05328/) (2016)
- [Improving Variational Inference with Inverse Autoregressive Flow](https://mlsanity.com/paper/1606.04934/) (2016)
- [Generating Sentences from a Continuous Space](https://mlsanity.com/paper/1511.06349/) (2015)
- [Importance Weighted Autoencoders](https://mlsanity.com/paper/1509.00519/) (2015)
- [Adversarial Autoencoders](https://mlsanity.com/paper/1511.05644/) (2015)
- [MADE: Masked Autoencoder for Distribution Estimation](https://mlsanity.com/paper/1502.03509/) (2015)
- [Style Transfer from Non-Parallel Text by Cross-Alignment](https://mlsanity.com/paper/1705.09655/) (2017)
- [Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing](https://mlsanity.com/paper/1903.10145/) (2019)
- [Variational Lossy Autoencoder](https://mlsanity.com/paper/1611.02731/) (2016)
- [Training generative neural networks via Maximum Mean Discrepancy optimization](https://mlsanity.com/paper/1505.03906/) (2015)
- [Variational Autoencoder for Deep Learning of Images, Labels and Captions](https://mlsanity.com/paper/1609.08976/) (2016)
- [Lagging Inference Networks and Posterior Collapse in Variational Autoencoders](https://mlsanity.com/paper/1901.05534/) (2019)
- [Preventing Posterior Collapse with delta-VAEs](https://mlsanity.com/paper/1901.03416/) (2019)
- [Semi-Amortized Variational Autoencoders](https://mlsanity.com/paper/1802.02550/) (2018)
- [Improved Variational Autoencoders for Text Modeling using Dilated Convolutions](https://mlsanity.com/paper/1702.08139/) (2017)
- [A Hybrid Convolutional Variational Autoencoder for Text Generation](https://mlsanity.com/paper/1702.02390/) (2017)
- [Implicit Deep Latent Variable Models for Text Generation](https://mlsanity.com/paper/1908.11527/) (2019)
- [Learning Representations by Maximizing Mutual Information in Variational Autoencoders](https://mlsanity.com/paper/1912.13361/) (2019)
- [A Batch Normalized Inference Network Keeps the KL Vanishing Away](https://mlsanity.com/paper/2004.12585/) (2020)
- [MAE: Mutual Posterior-Divergence Regularization for Variational AutoEncoders](https://mlsanity.com/paper/1901.01498/) (2019)
- [Collaborative Company Profiling: Insights from an Employee's Perspective](https://mlsanity.com/paper/1712.02987/) (2017)
- [Measuring the Popularity of Job Skills in Recruitment Market: A Multi-Criteria Approach](https://mlsanity.com/paper/1712.03087/) (2017)
- [Understanding VAEs in Fisher-Shannon Plane](https://mlsanity.com/paper/1807.03723/) (2018)

## Built on by (most cited first, 6 of 6)

- [Rethinking the Spatial Inconsistency in Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2404.05384/) (2024)
- [Conditional Variational Autoencoder for Sign Language Translation with Cross-Modal Alignment](https://mlsanity.com/paper/2312.15645/) (2023)
- [ReliCD: A Reliable Cognitive Diagnosis Framework with Confidence Awareness](https://mlsanity.com/paper/2401.10749/) (2023)
- [Job-SDF: A Multi-Granularity Dataset for Job Skill Demand Forecasting and Benchmarking](https://mlsanity.com/paper/2406.11920/) (2024)
- [Model-Based Counterfactual Explanations Incorporating Feature Space Attributes for Tabular Data](https://mlsanity.com/paper/2404.13224/) (2024)
- [On Feature Diversity in Energy-based Models](https://mlsanity.com/paper/2306.01489/) (2023)

## Abstract (arXiv)

As one of the most popular generative models, Variational Autoencoder (VAE) approximates the posterior of latent variables based on amortized variational inference. However, when the decoder network is sufficiently expressive, VAE may lead to posterior collapse; that is, uninformative latent representations may be learned. To this end, in this paper, we propose an alternative model, DU-VAE, for learning a more Diverse and less Uncertain latent space, and thus the representation can be learned in a meaningful and compact manner. Specifically, we first theoretically demonstrate that it will result in better latent space with high diversity and low uncertainty awareness by controlling the distribution of posterior's parameters across the whole data accordingly. Then, without the introduction of new loss terms or modifying training strategies, we propose to exploit Dropout on the variances and Batch-Normalization on the means simultaneously to regularize their distributions implicitly. Furthermore, to evaluate the generalization effect, we also exploit DU-VAE for inverse autoregressive flow based-VAE (VAE-IAF) empirically. Finally, extensive experiments on three benchmark datasets clearly show that our approach can outperform state-of-the-art baselines on both likelihood estimation and underlying classification tasks.