# Beyond Sinusoids: A Morlet Wavelet Framework for Transformer Positional Encoding

arXiv: https://arxiv.org/abs/2606.01258 · HTML: https://mlsanity.com/paper/2606.01258/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 12 references that are in the map.
- First posted on arXiv 31 May 2026.

## Builds on (most cited first, 9 of 9)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [FNet: Mixing Tokens with Fourier Transforms](https://mlsanity.com/paper/2105.03824/) (2021)
- [Language Through a Prism: A Spectral Approach for Multiscale Language Representations](https://mlsanity.com/paper/2011.04823/) (2020)
- [Towards Signal Processing In Large Language Models](https://mlsanity.com/paper/2406.10254/) (2024)
- [Energy-Gated Attention: Spectral Salience as an Inductive Bias for Transformer Attention](https://mlsanity.com/paper/2605.21842/) (2026)
- [Energy-Gated Attention and Wavelet Positional Encoding: Complementary Inductive Biases for Transformer Attention](https://mlsanity.com/paper/2605.26355/) (2026)

## Built on by (most cited first, 3 of 3)

- [Multiscale POD of Transformer Attention Fields: Scale-Selective Analysis via Morlet Scalogram](https://mlsanity.com/paper/2606.06573/) (2026)
- [FourierQK: Spectral Preprocessing of Query-Key Projections Improves Transformer Attention](https://mlsanity.com/paper/2607.07478/) (2026)
- [FourierQK: Filter Shape, Admissibility and the Leakage-Coverage Law](https://mlsanity.com/paper/2610.00009/) (2026)

## Abstract (arXiv)

Standard positional encodings for transformers - sinusoidal and rotary (RoPE) - treat every position as equally local: they encode where a token is, but not how far its positional influence should extend. We propose that the Morlet wavelet, which simultaneously minimises uncertainty in position and frequency, is the natural basis for positional encoding, and introduce Morlet Positional Encoding (MoPE): each embedding dimension learns its own frequency and locality bandwidth from data. The main theoretical result is a unification: sinusoidal PE and the RoPE correlation kernel both emerge as limiting cases of MoPE when locality is switched off (sigma_i -> infinity). The phase of MoPE recovers the RoPE rotation angle exactly; the amplitude adds a learned Gaussian locality kernel that standard encodings lack. Empirically, MoPE combined with Energy-Gated Attention achieves +0.119 improvement over standard attention on TinyShakespeare, outperforming either component alone. Analysis of the learned parameters reveals that all 128 frequency-bandwidth pairs converge to the wavelet admissibility boundary - an empirical observation consistent with a companion result on energy gating, suggesting a reproducible property of character-level language signals that warrants further investigation.