# Do traveling waves make good positional encodings?

arXiv: https://arxiv.org/abs/2511.11668 · HTML: https://mlsanity.com/paper/2511.11668/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 27 references that are in the map.
- First posted on arXiv 11 Nov 2025.

## Builds on (most cited first, 13 of 13)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Rotary Position Embedding for Vision Transformer](https://mlsanity.com/paper/2403.13298/) (2024)
- [On the Binding Problem in Artificial Neural Networks](https://mlsanity.com/paper/2012.05208/) (2020)
- [Round and Round We Go! What makes Rotary Positional Encodings useful?](https://mlsanity.com/paper/2410.06205/) (2024)
- [Complex-Valued Autoencoders for Object Discovery](https://mlsanity.com/paper/2204.02075/) (2022)
- [Artificial Kuramoto Oscillatory Neurons](https://mlsanity.com/paper/2410.13821/) (2024)
- [Rotating Features for Object Discovery](https://mlsanity.com/paper/2306.00600/) (2023)
- [Neuronal Synchrony in Complex-Valued Deep Networks](https://mlsanity.com/paper/1312.6115/) (2013)
- [Modeling Category-Selective Cortical Regions with Topographic Variational Autoencoders](https://mlsanity.com/paper/2110.13911/) (2021)
- [Computing with Residue Numbers in High-Dimensional Representation](https://mlsanity.com/paper/2311.04872/) (2023)
- [Topographic Deep Artificial Neural Networks (TDANNs) predict face selectivity topography in primate inferior temporal (IT) cortex](https://mlsanity.com/paper/1909.09847/) (2019)

## Abstract (arXiv)

Transformers rely on positional encoding to compensate for the inherent permutation invariance of self-attention. Traditional approaches use absolute sinusoidal embeddings or learned positional vectors, while more recent methods emphasize relative encodings to better capture translation equivariances. In this work, we propose RollPE, a novel positional encoding mechanism based on traveling waves, implemented by applying a circular roll operation to the query and key tensors in self-attention. This operation induces a relative shift in phase across positions, allowing the model to compute attention as a function of positional differences rather than absolute indices. We show this simple method significantly outperforms traditional absolute positional embeddings and is comparable to RoPE. We derive a continuous case of RollPE which implicitly imposes a topographic structure on the query and key space. We further derive a mathematical equivalence of RollPE to a particular configuration of RoPE. Viewing RollPE through the lens of traveling waves may allow us to simplify RoPE and relate it to processes of information flow in the brain.