mlsanityData to 14 Oct 2026

Latent Positional Information is in the Self-Attention Variance of Transformer Language Models Without Positional Embeddings

arXiv 2305.13571 ↗First posted 23 May 2023cs.CLcited by 23

Not on the map. Why?

Abstract (arXiv)

The use of positional embeddings in transformer language models is widely accepted. However, recent research has called into question the necessity of such embeddings. We further extend this inquiry by demonstrating that a randomly initialized and frozen transformer language model, devoid of positional embeddings, inherently encodes strong positional information through the shrinkage of self-attention variance. To quantify this variance, we derive the underlying distribution of each step within a transformer layer. Through empirical validation using a fully pretrained model, we show that the variance shrinkage effect still persists after extensive gradient updates. Our findings serve to justify the decision to discard positional embeddings and thus facilitate more efficient pretraining of transformer language models.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Builds on 17 of its 23 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

17 of 23 references.

References · 17

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  6. 2020On Layer Normalization in the Transformer Architecturecited by 922
  7. 2019Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernelcited by 213
  8. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 177
  9. 2020Probing Pretrained Language Models for Lexical Semanticscited by 156
  10. 2022Transformer Language Models without Positional Encodings Still Learn Positional Informationcited by 150
  11. 2019Language Modeling with Deep Transformerscited by 96
  12. 2022What Language Model to Train if You Have One Million GPU Hours?cited by 92
5 more
  1. 2020On the Computational Power of Transformers and its Implications in Sequence Modelingcited by 85
  2. 2020What's so special about BERT's layers? A closer look at the NLP pipeline in monolingual and multilingual modelscited by 35
  3. 2020Positional Artefacts Propagate Through Masked Language Model Embeddingscited by 35
  4. 2019Learning Internal Representations (COLT 1995)cited by 25
  5. 2019Assessing the Ability of Self-Attention Networks to Learn Word Ordercited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 6; 2025: 12; 2026: 4.

23 papers.

Sort:

By area:

  1. 2024Length Generalization of Causal Transformers without Position Encodingcited by 47
  2. 2023Length Extrapolation of Transformers: A Survey from the Perspective of Positional Encodingcited by 32
  3. 2025Sliding Window Attention Training for Efficient Large Language Modelscited by 24
  4. 2025SWAN-GPT: An Efficient and Scalable Approach for Long-Context Language Modelingcited by 19
  5. 2024Exploring Context Window of Large Language Models via Decomposed Positional Vectorscited by 18
  6. 2024ReAttention: Training-Free Infinite Context with Finite Attention Scopecited by 13
  7. 2025Position Bias Mitigates Position Bias:Mitigate Position Bias Through Inter-Position Knowledge Distillationcited by 9
  8. 2025AhaKV: Adaptive Holistic Attention-Driven KV Cache Eviction for Efficient Inference of Large Language Modelscited by 4
  9. 2024Mitigate Position Bias in Large Language Models via Scaling a Single Dimensioncited by 3
  10. 2025LEDiT: Your Length-Extrapolatable Diffusion Transformer without Positional Encodingcited by 3
  11. 2025Language Models, Graph Searching, and Supervision Adulteration: When More Supervision is Less and How to Make More Morecited by 3
  12. 2026Attention Needs to Focus: A Unified Perspective on Attention Allocationcited by 3
11 more
  1. 2026Same Architecture, Different Capacity: Optimizer-Induced Spectral Scaling Lawscited by 2
  2. 2025Beyond Isolated Capabilities: Bridging Long CoT Reasoning and Long-Context Understandingcited by 1
  3. 2025Positional Encoding via Token-Aware Phase Attentioncited by 1
  4. 2025LayerNorm Induces Recency Bias in Transformer Decoderscited by 1
  5. 2025TwIST: Rigging the Lottery in Transformers with Independent Subnetwork Trainingcited by 1
  6. 2024Breaking Symmetry When Training Transformerscited by 0
  7. 2024Is Smoothness the Key to Robustness? A Comparison of Attention and Convolution Models Using a Novel Metriccited by 0
  8. 2025Converting Transformers into DGNNs Formcited by 0
  9. 2025A Unified Geometric Space for Topological Alignment Between Transformer-Based Models and Human Brain Networkscited by 0
  10. 2026Temporal Context Reinstatement Drives Episodic-Like Order Memory in Long-Context Language Modelscited by 0
  11. 2026How Local Mixing Encodes Relative Position in Global NoPE Attentioncited by 0