# Transformer Learning of Chaotic Collective Dynamics in Many-Body Systems

arXiv: https://arxiv.org/abs/2601.19080 · HTML: https://mlsanity.com/paper/2601.19080/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 73 references that are in the map.
- First posted on arXiv 27 Jan 2026.

## Builds on (most cited first, 11 of 11)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [A Survey on Visual Transformer](https://mlsanity.com/paper/2012.12556/) (2020)
- [Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network](https://mlsanity.com/paper/1808.03314/) (2018)
- [A Critical Review of Recurrent Neural Networks for Sequence Learning](https://mlsanity.com/paper/1506.00019/) (2015)
- [Transformers for Modeling Physical Systems](https://mlsanity.com/paper/2010.03957/) (2020)
- [β-Variational autoencoders and transformers for reduced-order modelling of fluid flows](https://mlsanity.com/paper/2304.03571/) (2023)
- [ASTROMER: A transformer-based embedding for the representation of light curves](https://mlsanity.com/paper/2205.01677/) (2022)
- [Ab-initio variational wave functions for the time-dependent many-electron Schr"odinger equation](https://mlsanity.com/paper/2403.07447/) (2024)
- [Easy attention: A simple attention mechanism for temporal predictions with transformers](https://mlsanity.com/paper/2308.12874/) (2023)
- [Echo State network for coarsening dynamics of charge density waves](https://mlsanity.com/paper/2412.11982/) (2024)

## Abstract (arXiv)

Learning reduced descriptions of chaotic many-body dynamics is fundamentally challenging: although microscopic equations are Markovian, collective observables exhibit strong memory and exponential sensitivity to initial conditions and prediction errors. We show that a self-attention-based transformer framework provides an effective approach for modeling such chaotic collective dynamics directly from time-series data. By selectively reweighting long-range temporal correlations, the transformer learns a non-Markovian reduced description that overcomes intrinsic limitations of conventional recurrent architectures. As a concrete demonstration, we study the one-dimensional semiclassical Holstein model, where interaction quenches induce strongly nonlinear and chaotic dynamics of the charge-density-wave order parameter. While pointwise predictions inevitably diverge at long times, the transformer faithfully reproduces the statistical "climate" of the chaos, including temporal correlations and characteristic decay scales. Our results establish self-attention as a powerful mechanism for learning effective reduced dynamics in chaotic many-body systems.