# Position Information Emerges in Causal Transformers Without Positional Encodings via Similarity of Nearby Embeddings

arXiv: https://arxiv.org/abs/2501.00073 · HTML: https://mlsanity.com/paper/2501.00073/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Dec 2024.

## Built on by (most cited first, 8 of 8)

- [Understanding Task Vectors in In-Context Learning: Emergence, Functionality, and Limitations](https://mlsanity.com/paper/2506.09048/) (2025)
- [Deconstructing Positional Information: From Attention Logits to Training Biases](https://mlsanity.com/paper/2505.13027/) (2025)
- [DoPE: Denoising Rotary Position Embedding](https://mlsanity.com/paper/2511.09146/) (2025)
- [Efficient Autoregressive Inference for Transformer Probabilistic Models](https://mlsanity.com/paper/2510.09477/) (2025)
- [LayerNorm Induces Recency Bias in Transformer Decoders](https://mlsanity.com/paper/2509.21042/) (2025)
- [NVIDIA Nemotron Parse 1.1](https://mlsanity.com/paper/2511.20478/) (2025)
- [Temporal Context Reinstatement Drives Episodic-Like Order Memory in Long-Context Language Models](https://mlsanity.com/paper/2607.22575/) (2026)
- [How Local Mixing Encodes Relative Position in Global NoPE Attention](https://mlsanity.com/paper/2609.38109/) (2026)

## Abstract (arXiv)

Transformers with causal attention can solve tasks that require positional information without using positional encodings. In this work, we propose and investigate a new hypothesis about how positional information can be stored without using explicit positional encoding. We observe that nearby embeddings are more similar to each other than faraway embeddings, allowing the transformer to potentially reconstruct the positions of tokens. We show that this pattern can occur in both the trained and the randomly initialized Transformer models with causal attention and no positional encodings over a common range of hyperparameters.