mlsanityData to 14 Oct 2026

Rethinking PCA Through Duality

arXiv 2510.18130 ↗First posted 20 Oct 2025cs.LG math.OC stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

Motivated by the recently shown connection between self-attention and (kernel) principal component analysis (PCA), we revisit the fundamentals of PCA. Using the difference-of-convex (DC) framework, we present several novel formulations and provide new theoretical insights. In particular, we show the kernelizability and out-of-sample applicability for a PCA-like family of problems. Moreover, we uncover that simultaneous iteration, which is connected to the classical QR algorithm, is an instance of the difference-of-convex algorithm (DCA), offering an optimization perspective on this longstanding method. Further, we describe new algorithms for PCA and empirically compare them with state-of-the-art methods. Lastly, we introduce a kernelizable dual formulation for a robust variant of PCA that minimizes the l1 deviation of the reconstruction errors.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 24 of its 82 references that are in the map. First posted on arXiv 20 Oct 2025.

Builds on

24 of 82 references.

References · 24

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  8. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  9. 2021End-to-end Audio-visual Speech Recognition with Conformerscited by 151
  10. 2021Is Attention Better Than Matrix Decomposition?cited by 77
  11. 2022Squeezeformer: An Efficient Transformer for Automatic Speech Recognitioncited by 51
  12. 2018From Principal Subspaces to Principal Components with Linear Autoencoderscited by 39
12 more
  1. 2018An Overview of Robust Subspace Recoverycited by 29
  2. 2023Primal-Attention: Self-attention through Asymmetric Kernel SVD in Primal Representationcited by 29
  3. 2018Sparse Principal Component Analysis via Variable Projectioncited by 9
  4. 2016Forward-backward envelope for the sum of two nonconvex functions: Further properties and nonmonotone line-search algorithmscited by 8
  5. 2018Exactly Robust Kernel Principal Component Analysiscited by 7
  6. 2024Unveiling the Hidden Structure of Self-Attention via Kernel Principal Component Analysiscited by 7
  7. 2023Deep Kernel Principal Component Analysis for Multi-level Feature Learningcited by 6
  8. 2025Tight Analysis of Difference-of-Convex Algorithm (DCA) Improves Convergence Rates for Proximal Gradient Descentcited by 3
  9. 2025A New Pair of GloVescited by 3
  10. 2023Extending Kernel PCA through Dualization: Sparsity, Robustness and Fast Algorithmscited by 2
  11. 2025Forward-backward splitting under the light of generalized convexitycited by 2
  12. 2023Gradient-type subspace iteration methods for the symmetric eigenvalue problemcited by 1

Built on by

Not cited yet.