mlsanityData to 14 Oct 2026

On Vanishing Variance in Transformer Length Generalization

arXiv 2504.02827 ↗First posted 3 Apr 2025cs.LG cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

It is a widely known issue that Transformers, when trained on shorter sequences, fail to generalize robustly to longer ones at test time. This raises the question of whether Transformer models are real reasoning engines, despite their impressive abilities in mathematical problem solving and code synthesis. In this paper, we offer a vanishing variance perspective on this issue. To the best of our knowledge, we are the first to demonstrate that even for today's frontier models, a longer sequence length results in a decrease in variance in the output of the multi-head attention modules. On the argmax retrieval and dictionary lookup tasks, our experiments show that applying layer normalization after the attention outputs leads to significantly better length generalization. Our analyses attribute this improvement to a reduction-though not a complete elimination-of the distribution shift caused by vanishing variance.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 27 of its 32 references that are in the map. First posted on arXiv 3 Apr 2025.

Builds on

27 of 32 references.

References · 27

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2022Scalable Diffusion Models with Transformerscited by 6,054
  6. 2016Layer Normalizationcited by 5,669
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  8. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  9. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  10. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  11. 2018Self-Attention with Relative Position Representationscited by 1,120
  12. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
15 more
  1. 2021Domain Generalization: A Surveycited by 764
  2. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  3. 2016RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanismcited by 349
  4. 2016From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classificationcited by 341
  5. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 327
  6. 2022Exploring Length Generalization in Large Language Modelscited by 206
  7. 2023What Algorithms can Transformers Learn? A Study in Length Generalizationcited by 199
  8. 2019Adaptively Sparse Transformerscited by 155
  9. 2022Transformer Language Models without Positional Encodings Still Learn Positional Informationcited by 150
  10. 2023Randomized Positional Encodings Boost Length Generalization of Transformerscited by 117
  11. 2022The CLRS Algorithmic Reasoning Benchmarkcited by 106
  12. 2022KERPLE: Kernelized Relative Positional Embedding for Length Extrapolationcited by 86
  13. 2024Transformers Can Achieve Length Generalization But Not Robustlycited by 86
  14. 2024Softmax is not Enough (for Sharp Size Generalisation)cited by 46
  15. 2024Scaling Stick-Breaking Attention: An Efficient Implementation and In-depth Studycited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

  1. 2025What Happens Next? Anticipating Future Motion by Generating Point Trajectoriescited by 10
  2. 2025Quantitative Bounds for Length Generalization in Transformerscited by 8
  3. 2026Instruct-Particulate: Scaling Feed-Forward 3D Object Articulation with Kinematic Controlcited by 4
  4. 2026Stabilizing Extrapolation in Looped Transformers via Learned Stochastic Stoppingcited by 4
  5. 2026Can Language Models Actually Retrieve In-Context? Drowning in Documents at Million Token Scalecited by 2
  6. 2026Refit the Probe: Single-Direction Ablation Is Not a Necessity Testcited by 0