mlsanityData to 14 Oct 2026

Unveiling the Hidden Structure of Self-Attention via Kernel Principal Component Analysis

arXiv 2406.13762 ↗First posted 19 Jun 2024cs.LG cs.AI cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

The remarkable success of transformers in sequence modeling tasks, spanning various applications in natural language processing and computer vision, is attributed to the critical role of self-attention. Similar to the development of most deep learning models, the construction of these attention mechanisms relies on heuristics and experience. In our work, we derive self-attention from kernel principal component analysis (kernel PCA) and show that self-attention projects its query vectors onto the principal component axes of its key matrix in a feature space. We then formulate the exact formula for the value matrix in self-attention, theoretically and empirically demonstrating that this value matrix captures the eigenvectors of the Gram matrix of the key vectors in self-attention. Leveraging our kernel PCA framework, we propose Attention with Robust Principal Components (RPC-Attention), a novel class of robust attention that is resilient to data contamination. We empirically demonstrate the advantages of RPC-Attention over softmax attention on the ImageNet-1K object classification, WikiText-103 language modeling, and ADE20K image segmentation task.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 71 of its 101 references that are in the map. First posted on arXiv 19 Jun 2024.

Builds on

71 of 101 references.

References · 71

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  8. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  9. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  12. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
38 more
  1. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  2. 2016Pointer Sentinel Mixture Modelscited by 3,096
  3. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
  4. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  5. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  6. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  7. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  8. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  9. 2020Rethinking Attention with Performerscited by 1,601
  10. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  11. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  12. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,451
  13. 2019Natural Adversarial Examplescited by 1,408
  14. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  15. 2021Transformers in Vision: A Surveycited by 1,013
  16. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  17. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  18. 2019Square Attack: a query-efficient black-box adversarial attack via random searchcited by 685
  19. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  20. 2019Designing and Interpreting Probes with Control Taskscited by 640
  21. 2019Deep Equilibrium Modelscited by 615
  22. 2017OptNet: Differentiable Optimization as a Layer in Neural Networkscited by 599
  23. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 502
  24. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 419
  25. 2017A Structured Self-attentive Sentence Embeddingcited by 318
  26. 2021Understanding Robustness of Transformers for Image Classificationcited by 249
  27. 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 243
  28. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  29. 2018Adversarial Risk and the Dangers of Evaluating Against Weak Attackscited by 221
  30. 2016A Decomposable Attention Model for Natural Language Inferencecited by 215
  31. 2022Multi-Game Decision Transformerscited by 214
  32. 2019Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernelcited by 213
  33. 2021Vision Transformers are Robust Learnerscited by 207
  34. 2022Online Decision Transformercited by 196
  35. 2019Adversarial Training and Robustness for Multiple Perturbationscited by 183
  36. 2023A mathematical perspective on Transformerscited by 164
  37. 2022Understanding The Robustness in Vision Transformerscited by 144
  38. 2023The emergence of clusters in self-attention dynamicscited by 140

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 2.

7 papers.

  1. 2025Tight Clusters Make Specialized Expertscited by 8
  2. 2025Graph-Aware Isomorphic Attention for Adaptive Dynamics in Transformerscited by 7
  3. 2025Transformer Meets Twicing: Harnessing Unattended Residual Informationcited by 3
  4. 2025Rethinking PCA Through Dualitycited by 0
  5. 2025LUNA: Linear Universal Neural Attention with Generalization Guaranteescited by 0
  6. 2026Self-Attention And Beyond the Infinite: Towards Linear Transformers with Infinite Self-Attentioncited by 0
  7. 2026Anti Mode-Collapse in Mean-Field Transformer via Auxiliary Variablescited by 0