mlsanityData to 14 Oct 2026

Variational Autoencoders Pursue PCA Directions (by Accident)

arXiv 1812.06775 ↗First posted 17 Dec 2018cs.LG cs.CV stat.MLcited by 85

Not on the map. Why?

Abstract (arXiv)

The Variational Autoencoder (VAE) is a powerful architecture capable of representation learning and generative modeling. When it comes to learning interpretable (disentangled) representations, VAE and its variants show unparalleled performance. However, the reasons for this are unclear, since a very particular alignment of the latent embedding is needed but the design of the VAE does not encourage it in any explicit way. We address this matter and offer the following explanation: the diagonal approximation in the encoder together with the inherent stochasticity force local orthogonality of the decoder. The local behavior of promoting both reconstruction and orthogonality matches closely how the PCA embedding is chosen. Alongside providing an intuitive understanding, we justify the statement with full theoretical analysis as well as with experiments.

Key facts. Cited by 85 papers in the map (data through 14 Oct 2026). First posted on arXiv 17 Dec 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 17; 2022: 19; 2023: 15; 2024: 14; 2025: 10; 2026: 10.

85 papers.

Sort:

By area:

  1. 2021Towards Causal Representation Learningcited by 209
  2. 2022Disentangled Representation Learningcited by 112
  3. 2022ZeroEGGS: Zero-shot Example-based Gesture Generation from Speechcited by 84
  4. 2021Unsupervised Learning of Compositional Energy Conceptscited by 71
  5. 2022Function Classes for Identifiable Nonlinear Independent Component Analysiscited by 53
  6. 2021Learning Disentangled Representations in the Imaging Domaincited by 46
  7. 2024AROMA: Preserving Spatial Structure for Latent PDE Modeling with Local Neural Fieldscited by 39
  8. 2022Disentanglement of Correlated Factors via Hausdorff Factorized Supportcited by 37
  9. 2023Discovering Causal Relations and Equations from Datacited by 36
  10. 2022Disentanglement with Biological Constraints: A Theory of Functional Cell Typescited by 30
  11. 2021Multi-Facet Clustering Variational Autoencoderscited by 28
  12. 2022Embrace the Gap: VAEs Perform Independent Mechanism Analysiscited by 28
38 more
  1. 2022Symmetry-Based Representations for Artificial and Biological General Intelligencecited by 26
  2. 2022Efficient-VDVAE: Less is morecited by 26
  3. 2022A Geometric Perspective on Variational Autoencoderscited by 24
  4. 2023Causal Triplet: An Open Challenge for Intervention-centric Causal Representation Learningcited by 22
  5. 2021I Don't Need u: Identifiable Non-Linear ICA Without Side Informationcited by 21
  6. 2021Local Disentanglement in Variational Auto-Encoders Using Jacobian L1 Regularizationcited by 17
  7. 20213D Shape Variational Autoencoder Latent Disentanglement via Mini-Batch Feature Swapping for Bodies and Facescited by 15
  8. 2022How do Variational Autoencoders Learn? Insights from Representational Similaritycited by 10
  9. 2023DAVA: Disentangling Adversarial Variational Autoencodercited by 8
  10. 2021Be More Active! Understanding the Differences between Mean and Sampled Representations of Variational Autoencoderscited by 7
  11. 2022Overlooked Implications of the Reconstruction Loss for VAE Disentanglementcited by 7
  12. 2023Beyond Vanilla Variational Autoencoders: Detecting Posterior Collapse in Conditional and Hierarchical Variational Autoencoderscited by 7
  13. 2025Robustness of Nonlinear Representation Learningcited by 7
  14. 2022Meta Reinforcement Learning with Finite Training Tasks -- a Density Estimation Approachcited by 6
  15. 2023Matching aggregate posteriors in the variational autoencodercited by 6
  16. 2021Demystifying Inductive Biases for β-VAE Based Architecturescited by 5
  17. 2021Disentangling Generative Factors of Physical Fields Using Variational Autoencoderscited by 5
  18. 2022Towards Unsupervised Content Disentanglement in Sentence Representations via Syntactic Rolescited by 5
  19. 2022FONDUE: an algorithm to find the optimal dimensionality of the latent representations of variational autoencoderscited by 5
  20. 2023SC-VAE: Sparse Coding-based Variational Autoencoder with Learned ISTAcited by 4
  21. 2023Disentanglement Learning via Topologycited by 4
  22. 2023Latent Noise Segmentation: How Neural Noise Leads to the Emergence of Segmentation and Groupingcited by 4
  23. 2024A Statistical Analysis of Wasserstein Autoencoders for Intrinsically Low-dimensional Datacited by 4
  24. 2024Balanced Neural ODEs: nonlinear model order reduction and Koopman operator approximationscited by 4
  25. 2024α-TCVAE: On the relationship between Disentanglement and Diversitycited by 4
  26. 2025Unsupervised learning for anticipating critical transitionscited by 4
  27. 2025Posterior Collapse as a Phase Transition in Variational Autoencoderscited by 4
  28. 2021Contrastive Mixture of Posteriors for Counterfactual Inference, Data Integration and Fairnesscited by 3
  29. 2021Manifold Matching via Deep Metric Learning for Generative Modelingcited by 3
  30. 2023How good are variational autoencoders at transfer learning?cited by 3
  31. 2024Disentangled Representation Learning with the Gromov-Monge Gapcited by 3
  32. 2024On the Convergence Analysis of Over-Parameterized Variational Autoencoders: A Neural Tangent Kernel Perspectivecited by 3
  33. 2022Rotated Digit Recognition by Variational Autoencoders with Fixed Output Distributionscited by 2
  34. 2022Evaluating Disentanglement in Generative Models Without Knowledge of Latent Factorscited by 2
  35. 2023Correcting Flaws in Common Disentanglement Metricscited by 2
  36. 2023Unscented Autoencodercited by 2
  37. 2023Ensuring Topological Data-Structure Preservation under Autoencoder Compression due to Latent Space Regularization in Gauss--Legendre nodescited by 2
  38. 2024Comparing the information content of probabilistic representation spacescited by 2

50 most cited of 85 All 85 (JSON)