# The Neglected Sibling: Isotropic Gaussian Posterior for VAE

arXiv: https://arxiv.org/abs/2110.07383 · HTML: https://mlsanity.com/paper/2110.07383/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 34 references that are in the map.
- First posted on arXiv 14 Oct 2021.

## Builds on (most cited first, 24 of 24)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Generating Sentences from a Continuous Space](https://mlsanity.com/paper/1511.06349/) (2015)
- [Importance Weighted Autoencoders](https://mlsanity.com/paper/1509.00519/) (2015)
- [Understanding disentangling in β-VAE](https://mlsanity.com/paper/1804.03599/) (2018)
- [Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing](https://mlsanity.com/paper/1903.10145/) (2019)
- [Lagging Inference Networks and Posterior Collapse in Variational Autoencoders](https://mlsanity.com/paper/1901.05534/) (2019)
- [Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space](https://mlsanity.com/paper/2004.04092/) (2020)
- [The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations](https://mlsanity.com/paper/1511.02301/) (2015)
- [Preventing Posterior Collapse with delta-VAEs](https://mlsanity.com/paper/1901.03416/) (2019)
- [Improved Variational Autoencoders for Text Modeling using Dilated Convolutions](https://mlsanity.com/paper/1702.08139/) (2017)
- [Tighter Variational Bounds are Not Necessarily Better](https://mlsanity.com/paper/1802.04537/) (2018)
- [Avoiding Latent Variable Collapse With Generative Skip Models](https://mlsanity.com/paper/1807.04863/) (2018)
- [A Surprisingly Effective Fix for Deep Latent Variable Modeling of Text](https://mlsanity.com/paper/1909.00868/) (2019)
- [StructVAE: Tree-structured Latent Variable Models for Semi-supervised Semantic Parsing](https://mlsanity.com/paper/1806.07832/) (2018)
- [Preventing Posterior Collapse with Levenshtein Variational Autoencoder](https://mlsanity.com/paper/2004.14758/) (2020)
- [Effective Estimation of Deep Generative Language Models](https://mlsanity.com/paper/1904.08194/) (2019)
- [On the Importance of the Kullback-Leibler Divergence Term in Variational Autoencoders for Text Generation](https://mlsanity.com/paper/1909.13668/) (2019)
- [Do sequence-to-sequence VAEs learn global features of sentences?](https://mlsanity.com/paper/2004.07683/) (2020)
- [A Cross-Sentence Latent Variable Model for Semi-Supervised Text Sequence Matching](https://mlsanity.com/paper/1906.01343/) (2019)
- [Combining Deep Generative Models and Multi-lingual Pretraining for Semi-supervised Document Classification](https://mlsanity.com/paper/2101.10717/) (2021)

## Abstract (arXiv)

Deep generative models have been widely used in several areas of NLP, and various techniques have been proposed to augment them or address their training challenges. In this paper, we propose a simple modification to Variational Autoencoders (VAEs) by using an Isotropic Gaussian Posterior (IGP) that allows for better utilisation of their latent representation space. This model avoids the sub-optimal behavior of VAEs related to inactive dimensions in the representation space. We provide both theoretical analysis, and empirical evidence on various datasets and tasks that show IGP leads to consistent improvement on several quantitative and qualitative grounds, from downstream task performance and sample efficiency to robustness. Additionally, we give insights about the representational properties encouraged by IGP and also show that its gain generalises to image domain as well.