# Robust Filter Attention: Self-Attention as Precision-Weighted State Estimation

arXiv: https://arxiv.org/abs/2509.04154 · HTML: https://mlsanity.com/paper/2509.04154/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 48 references that are in the map.
- First posted on arXiv 4 Sep 2025.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [Retentive Network: A Successor to Transformer for Large Language Models](https://mlsanity.com/paper/2307.08621/) (2023)
- [HiPPO: Recurrent Memory with Optimal Polynomial Projections](https://mlsanity.com/paper/2008.07669/) (2020)
- [Hopfield Networks is All You Need](https://mlsanity.com/paper/2008.02217/) (2020)
- [Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernel](https://mlsanity.com/paper/1908.11775/) (2019)
- [KalmanNet: Neural Network Aided Kalman Filtering for Partially Known Dynamics](https://mlsanity.com/paper/2107.10043/) (2021)
- [A Length-Extrapolatable Transformer](https://mlsanity.com/paper/2212.10554/) (2022)
- [Improving Transformers with Probabilistic Attention Keys](https://mlsanity.com/paper/2110.08678/) (2021)
- [Can a Transformer Represent a Kalman Filter?](https://mlsanity.com/paper/2312.06937/) (2023)
- [Kalman Filtering Attention for User Behavior Modeling in CTR Prediction](https://mlsanity.com/paper/2010.00985/) (2020)
- [ACE-NODE: Attentive Co-Evolving Neural Ordinary Differential Equations](https://mlsanity.com/paper/2105.14953/) (2021)
- [Probabilistic Attention for Interactive Segmentation](https://mlsanity.com/paper/2106.15338/) (2021)
- [Designing Robust Transformers using Robust Kernel Density Estimation](https://mlsanity.com/paper/2210.05794/) (2022)
- [Adaptive Kalman-Informed Transformer](https://mlsanity.com/paper/2401.09987/) (2024)
- [Revisiting Kernel Attention with Correlated Gaussian Process Representation](https://mlsanity.com/paper/2502.20525/) (2025)
- [Learning Self-Modulating Attention in Continuous Time Space with Applications to Sequential Recommendation](https://mlsanity.com/paper/2204.06517/) (2022)
- [Neural Extended Kalman Filters for Learning and Predicting Dynamics of Structural Systems](https://mlsanity.com/paper/2210.04165/) (2022)
- [Elliptical Attention](https://mlsanity.com/paper/2406.13770/) (2024)
- [Neural SDEs as a Unified Approach to Continuous-Domain Sequence Modeling](https://mlsanity.com/paper/2501.18871/) (2025)
- [Bayesian Attention Mechanism: A Probabilistic Framework for Positional Encoding and Context Length Extrapolation](https://mlsanity.com/paper/2505.22842/) (2025)
- [Nonlinear Kalman Filtering based on Self-Attention Mechanism and Lattice Trajectory Piecewise Linear Approximation](https://mlsanity.com/paper/2404.03915/) (2024)
- [Uncertainty Propagation Networks for Neural Ordinary Differential Equations](https://mlsanity.com/paper/2508.16815/) (2025)

## Built on by (most cited first, 1 of 1)

- [The Transformer as a Polar State Estimator](https://mlsanity.com/paper/2605.11007/) (2026)

## Abstract (arXiv)

We introduce Robust Filter Attention (RFA), a formulation of self-attention as a robust state estimator. Each token is treated as a noisy observation of a latent trajectory governed by a linear stochastic differential equation (SDE), and attention weights are determined by consistency under this model rather than static feature similarity. Under isotropic noise and decay assumptions, RFA matches the computational complexity of standard attention. On language modeling benchmarks, RFA achieves lower perplexity than RoPE within the training window while remaining stable under zero-shot extrapolation to longer contexts. The framework also provides a dynamical interpretation of standard positional mechanisms, connecting rotational embeddings and recency biases to transport and uncertainty propagation induced by dynamics.