# Rethinking PCA Through Duality

arXiv: https://arxiv.org/abs/2510.18130 · HTML: https://mlsanity.com/paper/2510.18130/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 82 references that are in the map.
- First posted on arXiv 20 Oct 2025.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Adaptive Input Representations for Neural Language Modeling](https://mlsanity.com/paper/1809.10853/) (2018)
- [End-to-end Audio-visual Speech Recognition with Conformers](https://mlsanity.com/paper/2102.06657/) (2021)
- [Is Attention Better Than Matrix Decomposition?](https://mlsanity.com/paper/2109.04553/) (2021)
- [Squeezeformer: An Efficient Transformer for Automatic Speech Recognition](https://mlsanity.com/paper/2206.00888/) (2022)
- [From Principal Subspaces to Principal Components with Linear Autoencoders](https://mlsanity.com/paper/1804.10253/) (2018)
- [An Overview of Robust Subspace Recovery](https://mlsanity.com/paper/1803.01013/) (2018)
- [Primal-Attention: Self-attention through Asymmetric Kernel SVD in Primal Representation](https://mlsanity.com/paper/2305.19798/) (2023)
- [Sparse Principal Component Analysis via Variable Projection](https://mlsanity.com/paper/1804.00341/) (2018)
- [Forward-backward envelope for the sum of two nonconvex functions: Further properties and nonmonotone line-search algorithms](https://mlsanity.com/paper/1606.06256/) (2016)
- [Exactly Robust Kernel Principal Component Analysis](https://mlsanity.com/paper/1802.10558/) (2018)
- [Unveiling the Hidden Structure of Self-Attention via Kernel Principal Component Analysis](https://mlsanity.com/paper/2406.13762/) (2024)
- [Deep Kernel Principal Component Analysis for Multi-level Feature Learning](https://mlsanity.com/paper/2302.11220/) (2023)
- [Tight Analysis of Difference-of-Convex Algorithm (DCA) Improves Convergence Rates for Proximal Gradient Descent](https://mlsanity.com/paper/2503.04486/) (2025)
- [A New Pair of GloVes](https://mlsanity.com/paper/2507.18103/) (2025)
- [Extending Kernel PCA through Dualization: Sparsity, Robustness and Fast Algorithms](https://mlsanity.com/paper/2306.05815/) (2023)
- [Forward-backward splitting under the light of generalized convexity](https://mlsanity.com/paper/2503.18098/) (2025)
- [Gradient-type subspace iteration methods for the symmetric eigenvalue problem](https://mlsanity.com/paper/2306.10379/) (2023)

## Abstract (arXiv)

Motivated by the recently shown connection between self-attention and (kernel) principal component analysis (PCA), we revisit the fundamentals of PCA. Using the difference-of-convex (DC) framework, we present several novel formulations and provide new theoretical insights. In particular, we show the kernelizability and out-of-sample applicability for a PCA-like family of problems. Moreover, we uncover that simultaneous iteration, which is connected to the classical QR algorithm, is an instance of the difference-of-convex algorithm (DCA), offering an optimization perspective on this longstanding method. Further, we describe new algorithms for PCA and empirically compare them with state-of-the-art methods. Lastly, we introduce a kernelizable dual formulation for a robust variant of PCA that minimizes the l1 deviation of the reconstruction errors.