mlsanityData to 14 Oct 2026

Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encoding

arXiv 2512.03058 ↗First posted 25 Nov 2025cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

This paper investigates the dynamical properties of tokens in pre-trained Transformer models and explores their application to improving Transformers. To this end, we analyze the dynamical system governing the continuous-time limit of the pre-trained model and characterize the asymptotic behavior of its solutions. Specifically, we characterize when tokens move closer to or farther from one another over time, depending on the model parameters. We provide sufficient conditions, based on these parameters, to identify scenarios where tokens either converge to zero or diverge to infinity. Unlike prior works, our conditions are broader in scope and more applicable to real-world models. Furthermore, we investigate how different forms of positional encoding -- specifically absolute and rotary -- affect these dynamical regimes. Empirical evidence reveals that the convergence scenario adversely impacts model performance. Motivated by these insights, we propose simple refinements to Transformer architectures that mitigate convergence behavior in models with absolute or rotary positional encoding. These findings support theoretical foundations and design principles for improving Transformer models.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 37 of its 57 references that are in the map. First posted on arXiv 25 Nov 2025.

Builds on

37 of 57 references.

References · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  10. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  11. 2016Pointer Sentinel Mixture Modelscited by 3,096
  12. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
25 more
  1. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  2. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  3. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  4. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  5. 2021Transformers in Vision: A Surveycited by 1,013
  6. 2020Efficient Transformers: A Surveycited by 790
  7. 2018Universal Transformerscited by 647
  8. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  9. 2021A Survey of Transformerscited by 401
  10. 2017A Structured Self-attentive Sentence Embeddingcited by 318
  11. 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 243
  12. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  13. 2016A Decomposable Attention Model for Natural Language Inferencecited by 215
  14. 2023A mathematical perspective on Transformerscited by 164
  15. 2023The emergence of clusters in self-attention dynamicscited by 140
  16. 2018Character-Level Language Modeling with Deeper Self-Attentioncited by 138
  17. 2019Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of Viewcited by 113
  18. 2018ResNet with one-neuron hidden layers is a Universal Approximatorcited by 79
  19. 2019Deep Learning via Dynamical Systems: An Approximation Perspectivecited by 78
  20. 2021Neural ODE control for classification, approximation and transportcited by 57
  21. 2019Approximation Capabilities of Neural ODEs and Invertible Residual Networkscited by 56
  22. 2024Dynamic metastability in the self-attention modelcited by 46
  23. 2021Redesigning the Transformer Architecture with Insights from Multi-particle Dynamical Systemscited by 31
  24. 2023Interpolation, Approximation and Controllability of Deep Neural Networkscited by 16
  25. 2024Clustering in pure-attention hardmax transformers and its role in sentiment analysiscited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Reachability and asymptotics of Gaussian Transformer dynamicscited by 1
  2. 2026Self-Attention Dynamics with Rotary Position Embeddings: Twisted States and Explicit Consensus Rates on the Spherecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024The Asymptotic Behavior of Attention in Transformers
  2. 2025On the Universality of Transformer Architectures; How Much Attention Is Enough?
  3. 2025From Condensation to Rank Collapse: A Two-Stage Analysis of Transformer Training Dynamics
  4. 2025TransXSSM: A Hybrid Transformer State Space Model with Unified Rotary Position Embedding
  5. 2025Pause Tokens Strictly Increase the Expressivity of Constant-Depth Transformers
  6. 2025Two-Scale Latent Dynamics for Recurrent-Depth Transformers
  7. 2025The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training
  8. 2026Rethinking the Role of Positional Encoding: Sliding-Window Transformers without PE Remain Turing Complete