# Can Transformers overcome the lack of data in the simulation of history-dependent flows?

arXiv: https://arxiv.org/abs/2512.16305 · HTML: https://mlsanity.com/paper/2512.16305/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 62 references that are in the map.
- First posted on arXiv 18 Dec 2025.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Scientific Machine Learning through Physics-Informed Neural Networks: Where we are and What's next](https://mlsanity.com/paper/2201.05624/) (2022)
- [Transformers in Time Series: A Survey](https://mlsanity.com/paper/2202.07125/) (2022)
- [Choose a Transformer: Fourier or Galerkin](https://mlsanity.com/paper/2105.14995/) (2021)
- [Transformer for Partial Differential Equations' Operator Learning](https://mlsanity.com/paper/2205.13671/) (2022)
- [Transformers for Modeling Physical Systems](https://mlsanity.com/paper/2010.03957/) (2020)
- [β-Variational autoencoders and transformers for reduced-order modelling of fluid flows](https://mlsanity.com/paper/2304.03571/) (2023)
- [A Practical Survey on Faster and Lighter Transformers](https://mlsanity.com/paper/2103.14636/) (2021)
- [Structure-preserving neural networks](https://mlsanity.com/paper/2004.04653/) (2020)
- [GFINNs: GENERIC Formalism Informed Neural Networks for Deterministic and Stochastic Dynamical Systems](https://mlsanity.com/paper/2109.00092/) (2021)
- [Deep learning of thermodynamics-aware reduced-order models from data](https://mlsanity.com/paper/2007.03758/) (2020)
- [Transformers as Neural Operators for Solutions of Differential Equations with Finite Regularity](https://mlsanity.com/paper/2405.19166/) (2024)
- [Physics-Guided, Physics-Informed, and Physics-Encoded Neural Networks in Scientific Computing](https://mlsanity.com/paper/2211.07377/) (2022)
- [Model Reduction with Memory and the Machine Learning of Dynamical Systems](https://mlsanity.com/paper/1808.04258/) (2018)
- [Efficiently Parameterized Neural Metriplectic Systems](https://mlsanity.com/paper/2405.16305/) (2024)
- [Data-driven parameterization of the generalized Langevin equation](https://mlsanity.com/paper/1606.02596/) (2016)
- [Physics-Enhanced Machine Learning: a position paper for dynamical systems investigations](https://mlsanity.com/paper/2405.05987/) (2024)
- [Thermodynamics of learning physical phenomena](https://mlsanity.com/paper/2207.12749/) (2022)
- [Lie-Poisson Neural Networks (LPNets): Data-Based Computing of Hamiltonian Systems with Symmetries](https://mlsanity.com/paper/2308.15349/) (2023)
- [Introducing Hybrid Modeling with Time-series-Transformers: A Comparative Study of Series and Parallel Approach in Batch Crystallization](https://mlsanity.com/paper/2308.05749/) (2023)
- [A comparison of Single- and Double-generator formalisms for Thermodynamics-Informed Neural Networks](https://mlsanity.com/paper/2404.01060/) (2024)
- [Volume-Preserving Transformers for Learning Time Series Data with Structure](https://mlsanity.com/paper/2312.11166/) (2023)
- [Thermodynamics-informed super-resolution of scarce temporal dynamics data](https://mlsanity.com/paper/2402.17506/) (2024)
- [Adversarial Disentanglement by Backpropagation with Physics-Informed Variational Autoencoder](https://mlsanity.com/paper/2506.13658/) (2025)

## Abstract (arXiv)

It is well known that the lack of information about certain variables necessary for the description of a dynamical system leads to the introduction of historical dependence (lack of Markovian character of the model) and noise. Traditionally, scientists have made up for these shortcomings by designing phenomenological variables that take into account this historical dependence (typically, conformational tensors in fluids). Often, these phenomenological variables are not easily measurable experimentally. In this work, we study to what extent Transformer architectures are able to cope with the lack of experimental data on these variables. The methodology is evaluated on three benchmark problems: a cylinder flow with no history dependence, a viscoelastic Couette flow modeled via the Oldroyd-B formalism, and a non-linear polymeric fluid described by the FENE model. Our results show that the Transformer outperforms a thermodynamically consistent, structure-preserving neural network with metriplectic bias in systems with missing experimental data, providing lower errors even in low-dimensional latent spaces. In contrast, for systems whose state variables can be fully known, the metriplectic model achieves superior performance.