# Variational autoencoders in the presence of low-dimensional data: landscape and implicit bias

arXiv: https://arxiv.org/abs/2112.06868 · HTML: https://mlsanity.com/paper/2112.06868/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 23 references that are in the map.
- First posted on arXiv 13 Dec 2021.

## Builds on (most cited first, 14 of 14)

- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [The Implicit Bias of Gradient Descent on Separable Data](https://mlsanity.com/paper/1710.10345/) (2017)
- [NVAE: A Deep Hierarchical Variational Autoencoder](https://mlsanity.com/paper/2007.03898/) (2020)
- [Implicit Regularization in Deep Matrix Factorization](https://mlsanity.com/paper/1905.13655/) (2019)
- [Implicit Regularization in Matrix Factorization](https://mlsanity.com/paper/1705.09280/) (2017)
- [Diagnosing and Enhancing VAE Models](https://mlsanity.com/paper/1903.05789/) (2019)
- [Gradient descent aligns the layers of deep linear networks](https://mlsanity.com/paper/1810.02032/) (2018)
- [Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learning](https://mlsanity.com/paper/2012.09839/) (2020)
- [Lagging Inference Networks and Posterior Collapse in Variational Autoencoders](https://mlsanity.com/paper/1901.05534/) (2019)
- [Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse](https://mlsanity.com/paper/1911.02469/) (2019)
- [Preventing Posterior Collapse with delta-VAEs](https://mlsanity.com/paper/1901.03416/) (2019)
- [The Usual Suspects? Reassessing Blame for VAE Posterior Collapse](https://mlsanity.com/paper/1912.10702/) (2019)
- [On Implicit Regularization in β-VAEs](https://mlsanity.com/paper/2002.00041/) (2020)

## Built on by (most cited first, 11 of 11)

- [Deep Generative Models through the Lens of the Manifold Hypothesis: A Survey and New Connections](https://mlsanity.com/paper/2404.02954/) (2024)
- [Exact Solutions of a Deep Linear Network](https://mlsanity.com/paper/2202.04777/) (2022)
- [Pursuit of a Discriminative Representation for Multiple Subspaces via Sequential Games](https://mlsanity.com/paper/2206.09120/) (2022)
- [A Statistical Analysis of Wasserstein Autoencoders for Intrinsically Low-dimensional Data](https://mlsanity.com/paper/2402.15710/) (2024)
- [Stochastic Neural Networks with Infinite Width are Deterministic](https://mlsanity.com/paper/2201.12724/) (2022)
- [On the Convergence Analysis of Over-Parameterized Variational Autoencoders: A Neural Tangent Kernel Perspective](https://mlsanity.com/paper/2409.05349/) (2024)
- [Improving Molecule Properties Through 2-Stage VAE](https://mlsanity.com/paper/2212.02750/) (2022)
- [Zero-Variance Gradients for Variational Autoencoders](https://mlsanity.com/paper/2508.03587/) (2025)
- [Objective-Agnostic Enhancement of Molecule Properties via Multi-Stage VAE](https://mlsanity.com/paper/2308.13066/) (2023)
- [Beta-Sigma VAE: Separating beta and decoder variance in Gaussian variational autoencoder](https://mlsanity.com/paper/2409.09361/) (2024)
- [Disentanglement as Identifiable Pushforward Factorisation](https://mlsanity.com/paper/2410.22559/) (2024)

## Abstract (arXiv)

Variational Autoencoders are one of the most commonly used generative models, particularly for image data. A prominent difficulty in training VAEs is data that is supported on a lower-dimensional manifold. Recent work by Dai and Wipf (2020) proposes a two-stage training algorithm for VAEs, based on a conjecture that in standard VAE training the generator will converge to a solution with 0 variance which is correctly supported on the ground truth manifold. They gave partial support for that conjecture by showing that some optima of the VAE loss do satisfy this property, but did not analyze the training dynamics. In this paper, we show that for linear encoders/decoders, the conjecture is true-that is the VAE training does recover a generator with support equal to the ground truth manifold-and does so due to an implicit bias of gradient descent rather than merely the VAE loss itself. In the nonlinear case, we show that VAE training frequently learns a higher-dimensional manifold which is a superset of the ground truth manifold.