# Factor Analysis, Probabilistic Principal Component Analysis, Variational Inference, and Variational Autoencoder: Tutorial and Survey arXiv: https://arxiv.org/abs/2101.00734 ยท HTML: https://mlsanity.com/paper/2101.00734/ ## Key facts - Cited by 21 papers in the map (data through 14 Oct 2026). - Builds on 21 of its 50 references that are in the map. - First posted on arXiv 4 Jan 2021. ## Builds on (most cited first, 21 of 21) - [NVAE: A Deep Hierarchical Variational Autoencoder](https://mlsanity.com/paper/2007.03898/) (2020) - [Tutorial on Variational Autoencoders](https://mlsanity.com/paper/1606.05908/) (2016) - [Variational Autoencoder for Deep Learning of Images, Labels and Captions](https://mlsanity.com/paper/1609.08976/) (2016) - [Non-convex Optimization for Machine Learning](https://mlsanity.com/paper/1712.07897/) (2017) - [Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks](https://mlsanity.com/paper/1701.04722/) (2017) - [An Uncertain Future: Forecasting from Static Images using Variational Autoencoders](https://mlsanity.com/paper/1606.07873/) (2016) - [Implicit Reparameterization Gradients](https://mlsanity.com/paper/1805.08498/) (2018) - [Towards Deeper Understanding of Variational Autoencoding Models](https://mlsanity.com/paper/1702.08658/) (2017) - [Eigenvalue and Generalized Eigenvalue Problems: Tutorial](https://mlsanity.com/paper/1903.11240/) (2019) - [Hamiltonian Variational Auto-Encoder](https://mlsanity.com/paper/1805.11328/) (2018) - [Variational Autoencoder with Implicit Optimal Priors](https://mlsanity.com/paper/1809.05284/) (2018) - [Tutorial: Deriving the Standard Variational Autoencoder (VAE) Loss Function](https://mlsanity.com/paper/1907.08956/) (2019) - [GLSR-VAE: Geodesic Latent Space Regularization for Variational AutoEncoder Architectures](https://mlsanity.com/paper/1707.04588/) (2017) - [Unsupervised and Supervised Principal Component Analysis: Tutorial](https://mlsanity.com/paper/1906.03148/) (2019) - [Sampling Algorithms, from Survey Sampling to Monte Carlo Methods: Tutorial and Literature Review](https://mlsanity.com/paper/2011.00901/) (2020) - [Generalized probabilistic principal component analysis of correlated data](https://mlsanity.com/paper/1808.10868/) (2018) - [Fitting A Mixture Distribution to Data: Tutorial](https://mlsanity.com/paper/1901.06708/) (2019) - [Stochastic Neighbor Embedding with Gaussian and Student-t Distributions: Tutorial and Survey](https://mlsanity.com/paper/2009.10301/) (2020) - [Variational Nonparametric Discriminant Analysis](https://mlsanity.com/paper/1812.03648/) (2018) - [Discovering Influential Factors in Variational Autoencoders](https://mlsanity.com/paper/1809.01804/) (2018) - [Nearly Optimal Variational Inference for High Dimensional Regression with Shrinkage Priors](https://mlsanity.com/paper/2010.12887/) (2020) ## Built on by (most cited first, 21 of 21) - [Spectral, Probabilistic, and Deep Metric Learning: Tutorial and Survey](https://mlsanity.com/paper/2201.09267/) (2022) - [Bridge the Inference Gaps of Neural Processes via Expectation Maximization](https://mlsanity.com/paper/2501.03264/) (2025) - [A Systematic Review of Federated Generative Models](https://mlsanity.com/paper/2405.16682/) (2024) - [REGLUE Your Latents with Global and Local Semantics for Entangled Diffusion](https://mlsanity.com/paper/2512.16636/) (2025) - [Generative Adversarial Networks and Adversarial Autoencoders: Tutorial and Survey](https://mlsanity.com/paper/2111.13282/) (2021) - [Estimating Direct and Indirect Causal Effects of Spatiotemporal Interventions in Presence of Spatial Interference](https://mlsanity.com/paper/2405.08174/) (2024) - [Calibrated Multimodal Representation Learning with Missing Modalities](https://mlsanity.com/paper/2511.12034/) (2025) - [On Philomatics and Psychomatics for Combining Philosophy and Psychology with Mathematics](https://mlsanity.com/paper/2308.13738/) (2023) - [NeuralFactors: A Novel Factor Learning Approach to Generative Modeling of Equities](https://mlsanity.com/paper/2408.01499/) (2024) - [Generative Locally Linear Embedding](https://mlsanity.com/paper/2104.01525/) (2021) - [Sufficient Dimension Reduction for High-Dimensional Regression and Low-Dimensional Embedding: Tutorial and Survey](https://mlsanity.com/paper/2110.09620/) (2021) - [Detecting and Mitigating DDoS Attacks with AI: A Survey](https://mlsanity.com/paper/2503.17867/) (2025) - [An Introduction to Discrete Variational Autoencoders](https://mlsanity.com/paper/2505.10344/) (2025) - [Theoretical Connection between Locally Linear Embedding, Factor Analysis, and Probabilistic PCA](https://mlsanity.com/paper/2203.13911/) (2022) - [Autocodificadores Variacionales (VAE) Fundamentos Te'oricos y Aplicaciones](https://mlsanity.com/paper/2302.09363/) (2023) - [Resetting a fixed broken ELBO](https://mlsanity.com/paper/2312.06828/) (2023) - [Filling in Missing FX Implied Volatilities with Uncertainties: Improving VAE-Based Volatility Imputation](https://mlsanity.com/paper/2411.05998/) (2024) - [Deep Generative and Discriminative Digital Twin endowed with Variational Autoencoder for Unsupervised Predictive Thermal Condition Monitoring of Physical Robots in Industry 6.0 and Society 6.0](https://mlsanity.com/paper/2509.12740/) (2025) - [A Systematic Framework for Tabular Data Disentanglement](https://mlsanity.com/paper/2604.07940/) (2026) - [Learning in Low-Dimensional Subspaces: Orthogonal Bottlenecks for Reinforcement Learning](https://mlsanity.com/paper/2605.26012/) (2026) - [Orthogonal Dendritic Intrinsic Networks: An Architecture for Significance-Ordered, Orthogonal Latent Spaces](https://mlsanity.com/paper/2607.05653/) (2026) ## Abstract (arXiv) This is a tutorial and survey paper on factor analysis, probabilistic Principal Component Analysis (PCA), variational inference, and Variational Autoencoder (VAE). These methods, which are tightly related, are dimensionality reduction and generative models. They assume that every data point is generated from or caused by a low-dimensional latent factor. By learning the parameters of distribution of latent space, the corresponding low-dimensional factors are found for the sake of dimensionality reduction. For their stochastic and generative behaviour, these models can also be used for generation of new data points in the data space. In this paper, we first start with variational inference where we derive the Evidence Lower Bound (ELBO) and Expectation Maximization (EM) for learning the parameters. Then, we introduce factor analysis, derive its joint and marginal distributions, and work out its EM steps. Probabilistic PCA is then explained, as a special case of factor analysis, and its closed-form solutions are derived. Finally, VAE is explained where the encoder, decoder and sampling from the latent space are introduced. Training VAE using both EM and backpropagation are explained.