# Kalman Linear Attention: Parallel Bayesian Filtering For Efficient Language Modelling and State Tracking arXiv: https://arxiv.org/abs/2602.10743 ยท HTML: https://mlsanity.com/paper/2602.10743/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Builds on 49 of its 71 references that are in the map. - First posted on arXiv 11 Feb 2026. ## Builds on (most cited first, 30 of 49) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019) - [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018) - [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023) - [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019) - [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022) - [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019) - [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021) - [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019) - [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018) - [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020) - [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019) - [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024) - [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018) - [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://mlsanity.com/paper/2406.17557/) (2024) - [The LAMBADA dataset: Word prediction requiring a broad discourse context](https://mlsanity.com/paper/1606.06031/) (2016) - [Simplified State Space Layers for Sequence Modeling](https://mlsanity.com/paper/2208.04933/) (2022) - [Reinforcement Learning with Deep Energy-Based Policies](https://mlsanity.com/paper/1702.08165/) (2017) - [Gated Delta Networks: Improving Mamba2 with Delta Rule](https://mlsanity.com/paper/2412.06464/) (2024) - [Gated Linear Attention Transformers with Hardware-Efficient Training](https://mlsanity.com/paper/2312.06635/) (2023) - [Linear Transformers Are Secretly Fast Weight Programmers](https://mlsanity.com/paper/2102.11174/) (2021) - [Diagonal State Spaces are as Effective as Structured State Spaces](https://mlsanity.com/paper/2203.14343/) (2022) - [xLSTM: Extended Long Short-Term Memory](https://mlsanity.com/paper/2405.04517/) (2024) - [Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images](https://mlsanity.com/paper/1506.07365/) (2015) - [Zoology: Measuring and Improving Recall in Efficient Language Models](https://mlsanity.com/paper/2312.04927/) (2023) - [Kimi Linear: An Expressive, Efficient Attention Architecture](https://mlsanity.com/paper/2510.26692/) (2025) - [The Parallelism Tradeoff: Limitations of Log-Precision Transformers](https://mlsanity.com/paper/2207.00729/) (2022) - [The Illusion of State in State-Space Models](https://mlsanity.com/paper/2404.08819/) (2024) - [Structured Inference Networks for Nonlinear State Space Models](https://mlsanity.com/paper/1609.09869/) (2016) - [Deep Kalman Filters](https://mlsanity.com/paper/1511.05121/) (2015) ## Built on by (most cited first, 8 of 8) - [Learning State-Tracking from Code Using Linear RNNs](https://mlsanity.com/paper/2602.14814/) (2026) - [Memory for Large Language Models](https://mlsanity.com/paper/2607.25380/) (2026) - [Kalman Delta Networks: Uncertainty-aware Associative Memory](https://mlsanity.com/paper/2609.07816/) (2026) - [The Transformer as a Polar State Estimator](https://mlsanity.com/paper/2605.11007/) (2026) - [Memory by Design: Bayesian Layers for Sequence Modeling](https://mlsanity.com/paper/2605.31163/) (2026) - [How Wrong Can a Good Predictor Be? Diverging Updates with Vanishing Predictive KL](https://mlsanity.com/paper/2609.11132/) (2026) - [Riccati State Space Models: Non-iterative Parallelization for Nonlinear Sequence Modeling](https://mlsanity.com/paper/2609.35441/) (2026) - [Voltic: Distinguishing Volatility from Stochasticity in Recurrent Memory](https://mlsanity.com/paper/2610.05700/) (2026) ## Abstract (arXiv) State-space language models such as Mamba and gated linear attention (GLA) offer linear-complexity, parallelisable alternatives to transformers, but their linear state updates limit expressivity and robust state tracking. We close this gap from a probabilistic angle, casting sequence mixing as exact Bayesian filtering with the Kalman filter as the core primitive. Classical Kalman filters give principled state and uncertainty estimates but are viewed as inherently sequential; we show that reparameterising them in information form turns their updates into an associative scan - so the per-token recurrent update is non-linear (a M"obius/precision recursion) yet remains temporally parallel. The resulting Kalman Linear Attention (KLA) layer is a drop-in sequence mixer that performs time-parallel probabilistic inference, carries an explicit belief-state uncertainty, and is strictly more expressive than GLA-style linear updates at the same computational cost. This expressivity translates directly into stronger state tracking: KLA solves permutation-composition (A5) tasks that linear SSMs and attention cannot, while staying scan-parallel. As a drop-in primitive it also matches or improves on modern SSMs and GLAs across synthetic token-manipulation and zero-shot commonsense benchmarks, and is among the first stacked Bayesian-filtering primitives trained at the billion-token scale.