mlsanityData to 14 Oct 2026

Can a Transformer Represent a Kalman Filter?

arXiv 2312.06937 ↗First posted 12 Dec 2023cs.LG stat.MLcited by 20

Not on the map. Why?

Abstract (arXiv)

Transformers are a class of autoregressive deep learning architectures which have recently achieved state-of-the-art performance in various vision, language, and robotics tasks. We revisit the problem of Kalman Filtering in linear dynamical systems and show that Transformers can approximate the Kalman Filter in a strong sense. Specifically, for any observable LTI system we construct an explicit causally-masked Transformer which implements the Kalman Filter, up to a small additive error which is bounded uniformly in time; we call our construction the Transformer Filter. Our construction is based on a two-step reduction. We first show that a softmax self-attention block can exactly represent a Nadaraya-Watson kernel smoothing estimator with a Gaussian kernel. We then show that this estimator closely approximates the Kalman Filter. We also investigate how the Transformer Filter can be used for measurement-feedback control and prove that the resulting nonlinear controllers closely approximate the performance of standard optimal control policies such as the LQG controller.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Builds on 9 of its 12 references that are in the map. First posted on arXiv 12 Dec 2023.

Builds on

9 of 12 references.

References · 9

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  4. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  5. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  6. 2022Online Decision Transformercited by 196
  7. 2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 139
  8. 2023Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretrainingcited by 83
  9. 2022Formal Algorithms for Transformerscited by 72

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 8; 2026: 7.

20 papers.

Sort:

By area:

  1. 2025Test-time regression: a unifying framework for designing sequence models with associative memorycited by 74
  2. 2024Transformers Handle Endogeneity in In-Context Linear Regressioncited by 8
  3. 2024In-Context Learning of Linear Systems: Generalization Theory and Applications to Operator Learningcited by 6
  4. 2024BI-EqNO: Generalized Approximate Bayesian Inference with an Equivariant Neural Operator Frameworkcited by 6
  5. 2025In-Context Learning of Linear Dynamical Systems with Transformers: Approximation Bounds and Depth-Separationcited by 4
  6. 2024State space models, emergence, and ergodicity: How many parameters are needed for stable predictions?cited by 2
  7. 2024Transformers as Implicit State Estimators: In-Context Learning in Dynamical Systemscited by 2
  8. 2025Local Stability and Region of Attraction Analysis for Neural Network Feedback Systems under Positivity Constraintscited by 2
  9. 2025Robust Filter Attention: Self-Attention as Precision-Weighted State Estimationcited by 1
  10. 2025In-Context Learning for Non-Stationary MIMO Equalizationcited by 1
  11. 2026Adversarial Robustness of Deep State Space Models for Forecastingcited by 1
  12. 2026Transformers Can Implement Preconditioned Richardson Iteration for In-Context Gaussian Kernel Regressioncited by 1
8 more
  1. 2025Dual Filter: A Transformer-like Inference Architecture for Hidden Markov Modelscited by 0
  2. 2025Sailing Towards Zero-Shot State Estimation using Foundation Models Combined with a UKFcited by 0
  3. 2025Real-Time Performance Analysis of Multi-Fidelity Residual Physics-Informed Neural Process-Based State Estimation for Robotic Systemscited by 0
  4. 2026Model-Free Neural Filtering: A Comparison with Classical Filters in Nonlinear Systemscited by 0
  5. 2026The Transformer as a Polar State Estimatorcited by 0
  6. 2026Support-Conditioned Flow Matching Is Kernel Smoothingcited by 0
  7. 2026Transformer-like Inference from Optimal Controlcited by 0
  8. 2026Two-Layer Linear Auto-Regressive Models Estimate Latent Statescited by 0