# Can Transformers Learn Optimal Filtering for Unknown Systems?

arXiv: https://arxiv.org/abs/2308.08536 · HTML: https://mlsanity.com/paper/2308.08536/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 32 references that are in the map.
- First posted on arXiv 16 Aug 2023.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [What Can Transformers Learn In-Context? A Case Study of Simple Function Classes](https://mlsanity.com/paper/2208.01066/) (2022)
- [Transformers as Algorithms: Generalization and Stability in In-context Learning](https://mlsanity.com/paper/2301.07067/) (2023)
- [In-context Reinforcement Learning with Algorithm Distillation](https://mlsanity.com/paper/2210.14215/) (2022)
- [Online Learning for Time Series Prediction](https://mlsanity.com/paper/1302.6927/) (2013)
- [Online Learning of the Kalman Filter with Logarithmic Regret](https://mlsanity.com/paper/2002.05141/) (2020)
- [On-Line Learning of Linear Dynamical Systems: Exponential Forgetting in Kalman Filters](https://mlsanity.com/paper/1809.05870/) (2018)
- [Sample Complexity of Kalman Filtering for Unknown Systems](https://mlsanity.com/paper/1912.12309/) (2019)
- [Reduced-Order Nonlinear Observers via Contraction Analysis and Convex Optimization](https://mlsanity.com/paper/2012.06158/) (2020)
- [Kernel Least Mean Square with Adaptive Kernel Size](https://mlsanity.com/paper/1401.5899/) (2014)

## Built on by (most cited first, 16 of 16)

- [Fine-grained Analysis of In-context Linear Estimation: Data, Architecture, and Beyond](https://mlsanity.com/paper/2407.10005/) (2024)
- [From system models to class models: An in-context learning paradigm](https://mlsanity.com/paper/2308.13380/) (2023)
- [Rethinking Transformers in Solving POMDPs](https://mlsanity.com/paper/2405.17358/) (2024)
- [On the adaptation of in-context learners for system identification](https://mlsanity.com/paper/2312.04083/) (2023)
- [Enhanced Transformer architecture for in-context learning of dynamical systems](https://mlsanity.com/paper/2410.03291/) (2024)
- [The Duality of Generative AI and Reinforcement Learning in Robotics: A Review](https://mlsanity.com/paper/2410.16411/) (2024)
- [Manifold meta-learning for reduced-complexity neural system identification](https://mlsanity.com/paper/2504.11811/) (2025)
- [Decomposing Prediction Mechanisms for In-Context Recall](https://mlsanity.com/paper/2507.01414/) (2025)
- [Delay Independent Safe Control with Neural Networks: Positive Lur'e Certificates for Risk Aware Autonomy](https://mlsanity.com/paper/2510.06661/) (2025)
- [Model-Free Neural Filtering: A Comparison with Classical Filters in Nonlinear Systems](https://mlsanity.com/paper/2601.21266/) (2026)
- [Target Parameterization in Diffusion Models for Nonlinear Spatiotemporal System Identification](https://mlsanity.com/paper/2604.17566/) (2026)
- [ASIA: an Autonomous System Identification Agent](https://mlsanity.com/paper/2605.10480/) (2026)
- [Transformer-like Inference from Optimal Control](https://mlsanity.com/paper/2605.15608/) (2026)
- [Two-Layer Linear Auto-Regressive Models Estimate Latent States](https://mlsanity.com/paper/2606.12691/) (2026)
- [Variational meta-learning inference for low dimensional neural system identification](https://mlsanity.com/paper/2607.18965/) (2026)
- [In-Context Learning for System Identification: Probabilistic Formulation, Long-Sequence Handling, and Benchmark Validation](https://mlsanity.com/paper/2610.13411/) (2026)

## Abstract (arXiv)

Transformer models have shown great success in natural language processing; however, their potential remains mostly unexplored for dynamical systems. In this work, we investigate the optimal output estimation problem using transformers, which generate output predictions using all the past ones. Particularly, we train the transformer using various distinct systems and then evaluate the performance on unseen systems with unknown dynamics. Empirically, the trained transformer adapts exceedingly well to different unseen systems and even matches the optimal performance given by the Kalman filter for linear systems. In more complex settings with non-i.i.d. noise, time-varying dynamics, and nonlinear dynamics like a quadrotor system with unknown parameters, transformers also demonstrate promising results. To support our experimental findings, we provide statistical guarantees that quantify the amount of training data required for the transformer to achieve a desired excess risk. Finally, we point out some limitations by identifying two classes of problems that lead to degraded performance, highlighting the need for caution when using transformers for control and estimation.