# The Transformer as a Polar State Estimator

arXiv: https://arxiv.org/abs/2605.11007 · HTML: https://mlsanity.com/paper/2605.11007/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 34 references that are in the map.
- First posted on arXiv 10 May 2026.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [Retentive Network: A Successor to Transformer for Large Language Models](https://mlsanity.com/paper/2307.08621/) (2023)
- [HiPPO: Recurrent Memory with Optimal Polynomial Projections](https://mlsanity.com/paper/2008.07669/) (2020)
- [Hopfield Networks is All You Need](https://mlsanity.com/paper/2008.02217/) (2020)
- [Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernel](https://mlsanity.com/paper/1908.11775/) (2019)
- [A mathematical perspective on Transformers](https://mlsanity.com/paper/2312.10794/) (2023)
- [A Length-Extrapolatable Transformer](https://mlsanity.com/paper/2212.10554/) (2022)
- [nGPT: Normalized Transformer with Representation Learning on the Hypersphere](https://mlsanity.com/paper/2410.01131/) (2024)
- [PaTH Attention: Position Encoding via Accumulating Householder Transformations](https://mlsanity.com/paper/2505.16381/) (2025)
- [Can a Transformer Represent a Kalman Filter?](https://mlsanity.com/paper/2312.06937/) (2023)
- [LieRE: Lie Rotational Positional Encodings](https://mlsanity.com/paper/2406.10322/) (2024)
- [The invariant extended Kalman filter as a stable observer](https://mlsanity.com/paper/1410.1465/) (2014)
- [Probabilistic Attention for Interactive Segmentation](https://mlsanity.com/paper/2106.15338/) (2021)
- [Revisiting Kernel Attention with Correlated Gaussian Process Representation](https://mlsanity.com/paper/2502.20525/) (2025)
- [RiemannFormer: A Framework for Attention in Curved Spaces](https://mlsanity.com/paper/2506.07405/) (2025)
- [Kalman Linear Attention: Parallel Bayesian Filtering For Efficient Language Modelling and State Tracking](https://mlsanity.com/paper/2602.10743/) (2026)
- [Traveling Words: A Geometric Interpretation of Transformers](https://mlsanity.com/paper/2309.07315/) (2023)
- [Kalman Filters on Differentiable Manifolds](https://mlsanity.com/paper/2102.03804/) (2021)
- [Bayesian Attention Mechanism: A Probabilistic Framework for Positional Encoding and Context Length Extrapolation](https://mlsanity.com/paper/2505.22842/) (2025)
- [Robust Filter Attention: Self-Attention as Precision-Weighted State Estimation](https://mlsanity.com/paper/2509.04154/) (2025)

## Abstract (arXiv)

We show that the core components of the Transformer---attention, residual connections, and normalization---arise naturally from a single geometric state estimation problem. Modeling the latent state in polar coordinates naturally separates radial and hyperspherical dynamics, yielding a precision-weighted filtering procedure in which normalization enforces the hyperspherical constraint, attention aggregates directional evidence, and the residual connection implements an incremental state update. The standard Transformer block with rotary positional encodings is recovered by discarding the geometric correction terms of the resulting state estimator, showing that its architecture follows from the underlying estimation problem rather than from independent design choices. The proposed textitPolar Transformer retains these geometric corrections.