# A Mechanistic Analysis of Transformers for Dynamical Systems

arXiv: https://arxiv.org/abs/2512.21113 · HTML: https://mlsanity.com/paper/2512.21113/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 52 references that are in the map.
- First posted on arXiv 24 Dec 2025.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [Choose a Transformer: Fourier or Galerkin](https://mlsanity.com/paper/2105.14995/) (2021)
- [Deep Lagrangian Networks: Using Physics as Model Prior for Deep Learning](https://mlsanity.com/paper/1907.04490/) (2019)
- [Poseidon: Efficient Foundation Models for PDEs](https://mlsanity.com/paper/2405.19101/) (2024)
- [Transformers for Modeling Physical Systems](https://mlsanity.com/paper/2010.03957/) (2020)
- [Structure-preserving neural networks](https://mlsanity.com/paper/2004.04653/) (2020)
- [Transformers as Neural Operators for Solutions of Differential Equations with Finite Regularity](https://mlsanity.com/paper/2405.19166/) (2024)
- [PDE-Transformer: Efficient and Versatile Transformers for Physics Simulations](https://mlsanity.com/paper/2505.24717/) (2025)
- [Zero-shot forecasting of chaotic systems](https://mlsanity.com/paper/2409.15771/) (2024)
- [Understanding the differences in Foundation Models: Attention, State Space Models, and Recurrent Neural Networks](https://mlsanity.com/paper/2405.15731/) (2024)
- [True Zero-Shot Inference of Dynamical Systems Preserving Long-Term Statistics](https://mlsanity.com/paper/2505.13192/) (2025)
- [Defining Foundation Models for Computational Science: A Call for Clarity and Rigor](https://mlsanity.com/paper/2505.22904/) (2025)
- [Double Diffusion Maps and their Latent Harmonics for Scientific Computations in Latent Space](https://mlsanity.com/paper/2204.12536/) (2022)
- [Physics-guided Deep Markov Models for Learning Nonlinear Dynamical Systems with Uncertainty](https://mlsanity.com/paper/2110.08607/) (2021)
- [Interpretable learning of effective dynamics for multiscale systems](https://mlsanity.com/paper/2309.05812/) (2023)
- [Implementation and (Inverse Modified) Error Analysis for implicitly-templated ODE-nets](https://mlsanity.com/paper/2303.17824/) (2023)
- [Introducing Hybrid Modeling with Time-series-Transformers: A Comparative Study of Series and Parallel Approach in Batch Crystallization](https://mlsanity.com/paper/2308.05749/) (2023)
- [Physics Steering: Causal Control of Cross-Domain Concepts in a Physics Foundation Model](https://mlsanity.com/paper/2511.20798/) (2025)
- [Certified Invertibility in Neural Networks via Mixed-Integer Programming](https://mlsanity.com/paper/2301.11783/) (2023)
- [Learning Effective Dynamics across Spatio-Temporal Scales of Complex Flows](https://mlsanity.com/paper/2502.07990/) (2025)
- [Bayesian Inference for PDE-based Inverse Problems using the Optimization of a Discrete Loss](https://mlsanity.com/paper/2510.15664/) (2025)
- [Forecasting precipitation in the Arctic using probabilistic machine learning informed by causal climate drivers](https://mlsanity.com/paper/2510.24254/) (2025)

## Built on by (most cited first, 2 of 2)

- [Transformers for dynamical systems learn transfer operators in-context](https://mlsanity.com/paper/2602.18679/) (2026)
- [Equation-Free Digital Twins for Nonlinear Structural Dynamics](https://mlsanity.com/paper/2605.00950/) (2026)

## Abstract (arXiv)

Transformers are increasingly adopted for modeling and forecasting time-series, yet their internal mechanisms remain poorly understood from a dynamical systems perspective. In contrast to classical autoregressive and state-space models, which benefit from well-established theoretical foundations, Transformer architectures are typically treated as black boxes. This gap becomes particularly relevant as attention-based models are considered for general-purpose or zero-shot forecasting across diverse dynamical regimes. In this work, we do not propose a new forecasting model, but instead investigate the representational capabilities and limitations of single-layer Transformers when applied to dynamical data. Building on a dynamical systems perspective, we interpret causal self-attention as a linear, history-dependent recurrence and analyze how it processes temporal information. Through a series of linear and nonlinear case studies, we identify distinct operational regimes. For linear systems, we show that in the single-head attention-only setting, the convexity constraint imposed by softmax attention restricts the class of autoregressive operators that can be represented, leading to oversmoothing when the target dynamics require mixed-sign lag coefficients. For nonlinear systems under partial observability, attention instead acts as an adaptive delay-embedding mechanism, enabling effective state reconstruction when sufficient temporal context and latent dimensionality are available. These results help bridge empirical observations with classical dynamical systems theory, providing insight into when and why Transformers succeed or fail as models of dynamical systems.