# Unsupervised and Supervised Principal Component Analysis: Tutorial

arXiv: https://arxiv.org/abs/1906.03148 · HTML: https://mlsanity.com/paper/1906.03148/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Jun 2019.

## Built on by (most cited first, 12 of 12)

- [Reproducing Kernel Hilbert Space, Mercer's Theorem, Eigenfunctions, Nystr"om Method, and Use of Kernels in Machine Learning: Tutorial and Survey](https://mlsanity.com/paper/2106.08443/) (2021)
- [Factor Analysis, Probabilistic Principal Component Analysis, Variational Inference, and Variational Autoencoder: Tutorial and Survey](https://mlsanity.com/paper/2101.00734/) (2021)
- [Spectral, Probabilistic, and Deep Metric Learning: Tutorial and Survey](https://mlsanity.com/paper/2201.09267/) (2022)
- [Laplacian-Based Dimensionality Reduction Including Spectral Clustering, Laplacian Eigenmap, Locality Preserving Projection, Graph Embedding, and Diffusion Map: Tutorial and Survey](https://mlsanity.com/paper/2106.02154/) (2021)
- [Johnson-Lindenstrauss Lemma, Linear and Nonlinear Random Projections, Random Fourier Features, and Random Kitchen Sinks: Tutorial and Survey](https://mlsanity.com/paper/2108.04172/) (2021)
- [PCA-based Multi Task Learning: a Random Matrix Approach](https://mlsanity.com/paper/2111.00924/) (2021)
- [Relating tSNE and UMAP to Classical Dimensionality Reduction](https://mlsanity.com/paper/2306.11898/) (2023)
- [On Manifold Hypothesis: Hypersurface Submanifold Embedding Using Osculating Hyperspheres](https://mlsanity.com/paper/2202.01619/) (2022)
- [Unified Framework for Spectral Dimensionality Reduction, Maximum Variance Unfolding, and Kernel Learning By Semidefinite Programming: Tutorial and Survey](https://mlsanity.com/paper/2106.15379/) (2021)
- [Sufficient Dimension Reduction for High-Dimensional Regression and Low-Dimensional Embedding: Tutorial and Survey](https://mlsanity.com/paper/2110.09620/) (2021)
- [Gravitational Dimensionality Reduction Using Newtonian Gravity and Einstein's General Relativity](https://mlsanity.com/paper/2211.01369/) (2022)
- [Interpretable Visualizations of Data Spaces for Classification Problems](https://mlsanity.com/paper/2503.05861/) (2025)

## Abstract (arXiv)

This is a detailed tutorial paper which explains the Principal Component Analysis (PCA), Supervised PCA (SPCA), kernel PCA, and kernel SPCA. We start with projection, PCA with eigen-decomposition, PCA with one and multiple projection directions, properties of the projection matrix, reconstruction error minimization, and we connect to autoencoder. Then, PCA with singular value decomposition, dual PCA, and kernel PCA are covered. SPCA using both scoring and Hilbert-Schmidt independence criterion are explained. Kernel SPCA using both direct and dual approaches are then introduced. We cover all cases of projection and reconstruction of training and out-of-sample data. Finally, some simulations are provided on Frey and AT&T face datasets for verifying the theory in practice.