# Variational Autoencoders Pursue PCA Directions (by Accident)

arXiv: https://arxiv.org/abs/1812.06775 · HTML: https://mlsanity.com/paper/1812.06775/

## Key facts

- Cited by 85 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Dec 2018.

## Built on by (most cited first, 30 of 85)

- [Towards Causal Representation Learning](https://mlsanity.com/paper/2102.11107/) (2021)
- [Disentangled Representation Learning](https://mlsanity.com/paper/2211.11695/) (2022)
- [ZeroEGGS: Zero-shot Example-based Gesture Generation from Speech](https://mlsanity.com/paper/2209.07556/) (2022)
- [Unsupervised Learning of Compositional Energy Concepts](https://mlsanity.com/paper/2111.03042/) (2021)
- [Function Classes for Identifiable Nonlinear Independent Component Analysis](https://mlsanity.com/paper/2208.06406/) (2022)
- [Learning Disentangled Representations in the Imaging Domain](https://mlsanity.com/paper/2108.12043/) (2021)
- [AROMA: Preserving Spatial Structure for Latent PDE Modeling with Local Neural Fields](https://mlsanity.com/paper/2406.02176/) (2024)
- [Disentanglement of Correlated Factors via Hausdorff Factorized Support](https://mlsanity.com/paper/2210.07347/) (2022)
- [Discovering Causal Relations and Equations from Data](https://mlsanity.com/paper/2305.13341/) (2023)
- [Disentanglement with Biological Constraints: A Theory of Functional Cell Types](https://mlsanity.com/paper/2210.01768/) (2022)
- [Multi-Facet Clustering Variational Autoencoders](https://mlsanity.com/paper/2106.05241/) (2021)
- [Embrace the Gap: VAEs Perform Independent Mechanism Analysis](https://mlsanity.com/paper/2206.02416/) (2022)
- [Symmetry-Based Representations for Artificial and Biological General Intelligence](https://mlsanity.com/paper/2203.09250/) (2022)
- [Efficient-VDVAE: Less is more](https://mlsanity.com/paper/2203.13751/) (2022)
- [A Geometric Perspective on Variational Autoencoders](https://mlsanity.com/paper/2209.07370/) (2022)
- [Causal Triplet: An Open Challenge for Intervention-centric Causal Representation Learning](https://mlsanity.com/paper/2301.05169/) (2023)
- [I Don't Need u: Identifiable Non-Linear ICA Without Side Information](https://mlsanity.com/paper/2106.05238/) (2021)
- [Local Disentanglement in Variational Auto-Encoders Using Jacobian L1 Regularization](https://mlsanity.com/paper/2106.02923/) (2021)
- [3D Shape Variational Autoencoder Latent Disentanglement via Mini-Batch Feature Swapping for Bodies and Faces](https://mlsanity.com/paper/2111.12448/) (2021)
- [How do Variational Autoencoders Learn? Insights from Representational Similarity](https://mlsanity.com/paper/2205.08399/) (2022)
- [DAVA: Disentangling Adversarial Variational Autoencoder](https://mlsanity.com/paper/2303.01384/) (2023)
- [Be More Active! Understanding the Differences between Mean and Sampled Representations of Variational Autoencoders](https://mlsanity.com/paper/2109.12679/) (2021)
- [Overlooked Implications of the Reconstruction Loss for VAE Disentanglement](https://mlsanity.com/paper/2202.13341/) (2022)
- [Beyond Vanilla Variational Autoencoders: Detecting Posterior Collapse in Conditional and Hierarchical Variational Autoencoders](https://mlsanity.com/paper/2306.05023/) (2023)
- [Robustness of Nonlinear Representation Learning](https://mlsanity.com/paper/2503.15355/) (2025)
- [Meta Reinforcement Learning with Finite Training Tasks -- a Density Estimation Approach](https://mlsanity.com/paper/2206.10716/) (2022)
- [Matching aggregate posteriors in the variational autoencoder](https://mlsanity.com/paper/2311.07693/) (2023)
- [Demystifying Inductive Biases for β-VAE Based Architectures](https://mlsanity.com/paper/2102.06822/) (2021)
- [Disentangling Generative Factors of Physical Fields Using Variational Autoencoders](https://mlsanity.com/paper/2109.07399/) (2021)
- [Towards Unsupervised Content Disentanglement in Sentence Representations via Syntactic Roles](https://mlsanity.com/paper/2206.11184/) (2022)

## Abstract (arXiv)

The Variational Autoencoder (VAE) is a powerful architecture capable of representation learning and generative modeling. When it comes to learning interpretable (disentangled) representations, VAE and its variants show unparalleled performance. However, the reasons for this are unclear, since a very particular alignment of the latent embedding is needed but the design of the VAE does not encourage it in any explicit way. We address this matter and offer the following explanation: the diagonal approximation in the encoder together with the inherent stochasticity force local orthogonality of the decoder. The local behavior of promoting both reconstruction and orthogonality matches closely how the PCA embedding is chosen. Alongside providing an intuitive understanding, we justify the statement with full theoretical analysis as well as with experiments.