# Precision Tracked Transformer via Kalman Filtering, Kriging and Process Noise

arXiv: https://arxiv.org/abs/2605.18832 · HTML: https://mlsanity.com/paper/2605.18832/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 33 references that are in the map.
- First posted on arXiv 12 May 2026.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [Unsupervised Dense Information Retrieval with Contrastive Learning](https://mlsanity.com/paper/2112.09118/) (2021)
- [Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/1808.09781/) (2018)
- [BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer](https://mlsanity.com/paper/1904.06690/) (2019)
- [DivideMix: Learning with Noisy Labels as Semi-supervised Learning](https://mlsanity.com/paper/2002.07394/) (2020)
- [TinyLlama: An Open-Source Small Language Model](https://mlsanity.com/paper/2401.02385/) (2024)
- [Massive Activations in Large Language Models](https://mlsanity.com/paper/2402.17762/) (2024)
- [Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free](https://mlsanity.com/paper/2505.06708/) (2025)
- [Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations](https://mlsanity.com/paper/2402.17152/) (2024)
- [KalmanNet: Neural Network Aided Kalman Filtering for Partially Known Dynamics](https://mlsanity.com/paper/2107.10043/) (2021)
- [Infinite attention: NNGP and NTK for deep attention networks](https://mlsanity.com/paper/2006.10540/) (2020)
- [Denoising Implicit Feedback for Recommendation](https://mlsanity.com/paper/2006.04153/) (2020)
- [Calibrating Transformers via Sparse Gaussian Processes](https://mlsanity.com/paper/2303.02444/) (2023)
- [Denoising Self-attentive Sequential Recommendation](https://mlsanity.com/paper/2212.04120/) (2022)
- [Attention Calibration for Transformer-based Sequential Recommendation](https://mlsanity.com/paper/2308.09419/) (2023)
- [Revisiting Kernel Attention with Correlated Gaussian Process Representation](https://mlsanity.com/paper/2502.20525/) (2025)
- [Elliptical Attention](https://mlsanity.com/paper/2406.13770/) (2024)
- [Self-Attention through Kernel-Eigen Pair Sparse Variational Gaussian Processes](https://mlsanity.com/paper/2402.01476/) (2024)

## Abstract (arXiv)

The Transformer is the foundational building block of modern AI, yet offers no principled handling of uncertainty, which is prevalent in real applications: cold-start tokens with sparse histories in sequential recommendation, heterogeneous signal quality in language models, and attention sinks induced by unconstrained softmax. Every token is treated with uniform confidence. We show this uniformity is a degenerate case of our Bayesian Filtering Transformer (BFT): attention becomes precision-weighted kriging, the residual connection becomes a Kalman update with adaptive gain, and the FFN becomes a dynamics model propagating precision via a Jacobian--plus--process-noise rule. Observation precision comes from a parameter-free Restricted Maximum Likelihood (REML) estimator with a conjugate Bayesian prior. BFT replaces any Transformer layer with negligible overhead. On sequential recommendation, BFT applied to three major architectures yields significant gains on six benchmarks, with the largest improvements on cold-start users and rare items where uncertainty is highest. On supervised fine-tuning of large language models with noisy data, BFT improves robustness in two regimes: noisy supervision (token-label corruption in question answering) and noisy context (retrieval-augmented QA with real RAG distractors). A single principled modification -- restoring precision -- unlocks substantial headroom across both classical sequence-modeling and modern LLM regimes.