mlsanityData to 14 Oct 2026

Asymptotic Behavior of Transformers

arXiv 2610.19358 ↗First posted 14 Oct 2026cs.LG cs.AI math.STcited by 0

Not on the map. Why?

Abstract (arXiv)

In this paper we study the behavior of transformer networks generating text over long periods of time. We show that even a three-token, single-layer attention-only transformer can fail to converge. We then study a rich class of transformers with arbitrary finite depth, layer normalization, MLPs, RoPE encodings, and an arbitrary mix of local and global layers. Every transformer in this class has a softmax temperature threshold above which its empirical cache distribution converges to a unique limit, independent of initial conditions. We show further that, while each transformer has a temperature threshold, the supremum of this set is infinite, so there is no global minimal temperature that forces all transformers to converge.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 34 of its 42 references that are in the map. First posted on arXiv 14 Oct 2026.

Builds on

34 of 42 references.

References · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  3. 2020Longformer: The Long-Document Transformercited by 3,413
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  5. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  6. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  7. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  8. 2018Self-Attention with Relative Position Representationscited by 1,120
  9. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  10. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  11. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  12. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
22 more
  1. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  2. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  3. 2020Hopfield Networks is All You Needcited by 475
  4. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 419
  5. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 327
  6. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 313
  7. 2022Memorizing Transformerscited by 226
  8. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 200
  9. 2023The emergence of clusters in self-attention dynamicscited by 140
  10. 2022A Length-Extrapolatable Transformercited by 127
  11. 2023Randomized Positional Encodings Boost Length Generalization of Transformerscited by 117
  12. 2024Base of RoPE Bounds Context Lengthcited by 52
  13. 2024Length Generalization of Causal Transformers without Position Encodingcited by 47
  14. 2024Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscapecited by 46
  15. 2024A Formal Framework for Understanding Length Generalization in Transformerscited by 45
  16. 2024Emergence of meta-stable clustering in mean-field transformer modelscited by 39
  17. 2025The Mean-Field Dynamics of Transformerscited by 30
  18. 2025Synchronization of mean-field models on the circlecited by 12
  19. 2026Symmetry Breaking in Transformers for Efficient and Interpretable Trainingcited by 5
  20. 2026Krause Synchronization Transformerscited by 5
  21. 2026A Structural Theory of Position Bias in Transformerscited by 5
  22. 2025Rotary Position Encodings for Graphscited by 2

Built on by

Not cited yet.