mlsanityData to 7 Oct 2026

Analogies between Transformer Layers and Power Method

arXiv 2605.25619 ↗First posted 25 May 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

In the paper we show that there is an analogy between the operations occurring in a layer of a transformer (projections and layer normalizations, disregarding the feedforward neural network) and a step in the power method. Coherently with this analogy, we show that passing through a layer the tokens tend to be tilted towards the principal eigenvector of a matrix which is the product of the output and value weight matrices of that layer. In the special case of a transformer with shared weights (i.e., in which all layers have identical weights) then the alignment with this principal eigenvector is particularly evident empirically, and can also be shown analytically. The analogy also suggests a method to steer the output of the transformer towards an arbitrary desired direction in token space.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 29 of its 35 references that are in the map. First posted on arXiv 25 May 2026.

Builds on

29 of 35 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  5. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  6. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  8. 2022Editing Models with Task Arithmeticcited by 1,076
  9. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  10. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
  11. 2023A mathematical perspective on Transformerscited by 160
  12. 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 154
17 more
  1. 2021Sinkformers: Transformers with Doubly Stochastic Attentioncited by 136
  2. 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 136
  3. 2023The emergence of clusters in self-attention dynamicscited by 133
  4. 2024The Prompt Report: A Systematic Survey of Prompt Engineering Techniquescited by 115
  5. 2019Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of Viewcited by 108
  6. 2022Revisiting Over-smoothing in BERT from the Perspective of Graphcited by 71
  7. 2024On the Role of Attention Masks and LayerNorm in Transformerscited by 45
  8. 2021On the Expressive Power of Self-Attention Matricescited by 44
  9. 2024Dynamic metastability in the self-attention modelcited by 42
  10. 2023Mitigating Over-smoothing in Transformers via Regularized Nonlocal Functionalscited by 31
  11. 2024Residual Connections and Normalization Can Provably Prevent Oversmoothing in GNNscited by 31
  12. 2021Redesigning the Transformer Architecture with Insights from Multi-particle Dynamical Systemscited by 29
  13. 2024Setting the Record Straight on Transformer Oversmoothingcited by 19
  14. 2025Two failure modes of deep transformers and how to avoid them: a unified theory of signal propagation at initialisationcited by 15
  15. 2024The Asymptotic Behavior of Attention in Transformerscited by 12
  16. 2025Multistability of Self-Attention Dynamics in Transformerscited by 6
  17. 2025A Theoretical Study of (Hyper) Self-Attention through the Lens of Interactions: Representation, Training, Generalizationcited by 4

Built on by

Not cited yet.