# Latent Positional Information is in the Self-Attention Variance of Transformer Language Models Without Positional Embeddings

arXiv: https://arxiv.org/abs/2305.13571 · HTML: https://mlsanity.com/paper/2305.13571/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 23 references that are in the map.
- First posted on arXiv 23 May 2023.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernel](https://mlsanity.com/paper/1908.11775/) (2019)
- [Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little](https://mlsanity.com/paper/2104.06644/) (2021)
- [Probing Pretrained Language Models for Lexical Semantics](https://mlsanity.com/paper/2010.05731/) (2020)
- [Transformer Language Models without Positional Encodings Still Learn Positional Information](https://mlsanity.com/paper/2203.16634/) (2022)
- [Language Modeling with Deep Transformers](https://mlsanity.com/paper/1905.04226/) (2019)
- [What Language Model to Train if You Have One Million GPU Hours?](https://mlsanity.com/paper/2210.15424/) (2022)
- [On the Computational Power of Transformers and its Implications in Sequence Modeling](https://mlsanity.com/paper/2006.09286/) (2020)
- [What's so special about BERT's layers? A closer look at the NLP pipeline in monolingual and multilingual models](https://mlsanity.com/paper/2004.06499/) (2020)
- [Positional Artefacts Propagate Through Masked Language Model Embeddings](https://mlsanity.com/paper/2011.04393/) (2020)
- [Learning Internal Representations (COLT 1995)](https://mlsanity.com/paper/1911.05781/) (2019)
- [Assessing the Ability of Self-Attention Networks to Learn Word Order](https://mlsanity.com/paper/1906.00592/) (2019)

## Built on by (most cited first, 23 of 23)

- [Length Generalization of Causal Transformers without Position Encoding](https://mlsanity.com/paper/2404.12224/) (2024)
- [Length Extrapolation of Transformers: A Survey from the Perspective of Positional Encoding](https://mlsanity.com/paper/2312.17044/) (2023)
- [Sliding Window Attention Training for Efficient Large Language Models](https://mlsanity.com/paper/2502.18845/) (2025)
- [SWAN-GPT: An Efficient and Scalable Approach for Long-Context Language Modeling](https://mlsanity.com/paper/2504.08719/) (2025)
- [Exploring Context Window of Large Language Models via Decomposed Positional Vectors](https://mlsanity.com/paper/2405.18009/) (2024)
- [ReAttention: Training-Free Infinite Context with Finite Attention Scope](https://mlsanity.com/paper/2407.15176/) (2024)
- [Position Bias Mitigates Position Bias:Mitigate Position Bias Through Inter-Position Knowledge Distillation](https://mlsanity.com/paper/2508.15709/) (2025)
- [AhaKV: Adaptive Holistic Attention-Driven KV Cache Eviction for Efficient Inference of Large Language Models](https://mlsanity.com/paper/2506.03762/) (2025)
- [Mitigate Position Bias in Large Language Models via Scaling a Single Dimension](https://mlsanity.com/paper/2406.02536/) (2024)
- [LEDiT: Your Length-Extrapolatable Diffusion Transformer without Positional Encoding](https://mlsanity.com/paper/2503.04344/) (2025)
- [Language Models, Graph Searching, and Supervision Adulteration: When More Supervision is Less and How to Make More More](https://mlsanity.com/paper/2503.10542/) (2025)
- [Attention Needs to Focus: A Unified Perspective on Attention Allocation](https://mlsanity.com/paper/2601.00919/) (2026)
- [Same Architecture, Different Capacity: Optimizer-Induced Spectral Scaling Laws](https://mlsanity.com/paper/2605.21803/) (2026)
- [Beyond Isolated Capabilities: Bridging Long CoT Reasoning and Long-Context Understanding](https://mlsanity.com/paper/2507.14849/) (2025)
- [Positional Encoding via Token-Aware Phase Attention](https://mlsanity.com/paper/2509.12635/) (2025)
- [LayerNorm Induces Recency Bias in Transformer Decoders](https://mlsanity.com/paper/2509.21042/) (2025)
- [TwIST: Rigging the Lottery in Transformers with Independent Subnetwork Training](https://mlsanity.com/paper/2511.03983/) (2025)
- [Breaking Symmetry When Training Transformers](https://mlsanity.com/paper/2402.05969/) (2024)
- [Is Smoothness the Key to Robustness? A Comparison of Attention and Convolution Models Using a Novel Metric](https://mlsanity.com/paper/2410.17628/) (2024)
- [Converting Transformers into DGNNs Form](https://mlsanity.com/paper/2502.00585/) (2025)
- [A Unified Geometric Space for Topological Alignment Between Transformer-Based Models and Human Brain Networks](https://mlsanity.com/paper/2510.24342/) (2025)
- [Temporal Context Reinstatement Drives Episodic-Like Order Memory in Long-Context Language Models](https://mlsanity.com/paper/2607.22575/) (2026)
- [How Local Mixing Encodes Relative Position in Global NoPE Attention](https://mlsanity.com/paper/2609.38109/) (2026)

## Abstract (arXiv)

The use of positional embeddings in transformer language models is widely accepted. However, recent research has called into question the necessity of such embeddings. We further extend this inquiry by demonstrating that a randomly initialized and frozen transformer language model, devoid of positional embeddings, inherently encodes strong positional information through the shrinkage of self-attention variance. To quantify this variance, we derive the underlying distribution of each step within a transformer layer. Through empirical validation using a fully pretrained model, we show that the variance shrinkage effect still persists after extensive gradient updates. Our findings serve to justify the decision to discard positional embeddings and thus facilitate more efficient pretraining of transformer language models.