mlsanityData to 14 Oct 2026

Transformers for Modeling Physical Systems

arXiv 2010.03957 ↗First posted 4 Oct 2020cs.LG physics.comp-phcited by 106

Not on the map. Why?

Abstract (arXiv)

Transformers are widely used in natural language processing due to their ability to model longer-term dependencies in text. Although these models achieve state-of-the-art performance for many language related tasks, their applicability outside of the natural language processing field has been minimal. In this work, we propose the use of transformer models for the prediction of dynamical systems representative of physical phenomena. The use of Koopman based embeddings provide a unique and powerful method for projecting any dynamical system into a vector representation which can then be predicted by a transformer. The proposed model is able to accurately predict various dynamical systems and outperform classical methods that are commonly used in the scientific machine learning literature.

Key facts. Cited by 106 papers in the map (data through 14 Oct 2026). First posted on arXiv 4 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 8; 2022: 10; 2023: 20; 2024: 23; 2025: 26; 2026: 19.

106 papers.

Sort:

By area:

  1. 2022Transformer for Partial Differential Equations' Operator Learningcited by 283
  2. 2023Scalable Transformer for PDE Surrogate Modelingcited by 128
  3. 2021PhyCRNet: Physics-informed Convolutional-Recurrent Network for Solving Spatiotemporal PDEscited by 98
  4. 2022Predicting Physics in Mesh-reduced Space with Temporal Attentioncited by 84
  5. 2023β-Variational autoencoders and transformers for reduced-order modelling of fluid flowscited by 77
  6. 2023Benchmarking Autoregressive Conditional Diffusion Models for Turbulent Flow Simulationcited by 71
  7. 2021Encoding physics to learn reaction-diffusion processescited by 61
  8. 2022Efficient Learning of Mesh-Based Physical Simulation with BSMS-GNNcited by 45
  9. 2022Mitigating spectral bias for the multiscale operator learningcited by 44
  10. 2024SepLLM: Accelerate Large Language Models by Compressing One Segment into One Separatorcited by 41
  11. 2021Learning effective stochastic differential equations from microscopic simulations: linking stochastic numerics to deep learningcited by 39
  12. 2024Transformers as Neural Operators for Solutions of Differential Equations with Finite Regularitycited by 34
38 more
  1. 2024Out-of-Domain Generalization in Dynamical Systems Reconstructioncited by 33
  2. 2024HAMLET: Graph Transformer Neural Operator for Partial Differential Equationscited by 24
  3. 2022Neural Integral Equationscited by 23
  4. 2021Is attention to bounding boxes all you need for pedestrian action prediction?cited by 21
  5. 2023Physics Informed Token Transformer for Solving Partial Differential Equationscited by 21
  6. 2025True Zero-Shot Inference of Dynamical Systems Preserving Long-Term Statisticscited by 20
  7. 2024Almost-Linear RNNs Yield Highly Interpretable Symbolic Codes in Dynamical Systems Reconstructioncited by 19
  8. 2025Lost in Latent Space: An Empirical Study of Latent Diffusion Models for Physics Emulationcited by 19
  9. 2022Seeing the forest and the tree: Building representations of both individual and collective dynamics with transformerscited by 18
  10. 2024PhyMPGN: Physics-encoded Message Passing Graph Network for spatiotemporal PDE systemscited by 18
  11. 2024Generative Learning of the Solution of Parametric Partial Differential Equations Using Guided Diffusion Models and Virtual Observationscited by 16
  12. 2023Multi-scale Time-stepping of Partial Differential Equations with Transformerscited by 14
  13. 2024Vectorized Conditional Neural Fields: A Framework for Solving Time-dependent Parametric Partial Differential Equationscited by 12
  14. 2025Machine learning for modelling unstructured grid data in computational physics: a reviewcited by 12
  15. 2022LordNet: An Efficient Neural Network for Learning to Solve Parametric Partial Differential Equations without Simulated Datacited by 10
  16. 2024DAPE V2: Process Attention Score as Feature Map for Length Extrapolationcited by 10
  17. 2022Physics-informed Deep Super-resolution for Spatiotemporal Datacited by 9
  18. 2023Towards Complex Dynamic Physics System Simulation with Graph Neural ODEscited by 8
  19. 2023A generative model for surrogates of spatial-temporal wildfire nowcastingcited by 8
  20. 2021State estimation with limited sensors -- A deep learning based approachcited by 7
  21. 2021GrADE: A graph based data-driven solver for time-dependent nonlinear partial differential equationscited by 7
  22. 2024AI enhanced data assimilation and uncertainty quantification applied to Geological Carbon Storagecited by 7
  23. 2023InVAErt networks: a data-driven framework for model synthesis and identifiability analysiscited by 6
  24. 2024Generative Learning for Forecasting the Dynamics of Complex Systemscited by 6
  25. 2025Evaluating and Mitigating LLM-as-a-judge Bias in Communication Systemscited by 6
  26. 2021CKNet: A Convolutional Neural Network Based on Koopman Operator for Modeling Latent Dynamics from Pixelscited by 5
  27. 2023Quantifying uncertainty for deep learning based forecasting and flow-reconstruction using neural architecture search ensemblescited by 5
  28. 2024Latent Neural PDE Solver: a reduced-order modelling framework for partial differential equationscited by 5
  29. 2024Learning with SASQuaTCh: a Novel Variational Quantum Transformer Architecture with Kernel-Based Self-Attentioncited by 5
  30. 2025MetaSym: A Symplectic Meta-learning Framework for Physical Intelligencecited by 5
  31. 2025FD-Bench: A Modular and Fair Benchmark for Data-driven Fluid Simulationcited by 5
  32. 2025Transforming Calabi-Yau Constructions: Generating New Calabi-Yau Manifolds with Transformerscited by 5
  33. 2022DLKoopman: A deep learning software package for Koopman theorycited by 4
  34. 2023AttNS: Attention-Inspired Numerical Solving For Limited Data Scenarioscited by 4
  35. 2024Resolution invariant deep operator network for PDEs with complex geometriescited by 4
  36. 2025Augmented Invertible Koopman Autoencoder for long-term time series forecastingcited by 4
  37. 2021A deep learning modeling framework to capture mixing patterns in reactive-transport systemscited by 3
  38. 2023Easy attention: A simple attention mechanism for temporal predictions with transformerscited by 3

50 most cited of 106 All 106 (JSON)