mlsanityData to 7 Oct 2026

Multi-scale Time-stepping of Partial Differential Equations with Transformers

arXiv 2311.02225 ↗First posted 3 Nov 2023cs.LGcited by 14

Not on the map. Why?

Abstract (arXiv)

Developing fast surrogates for Partial Differential Equations (PDEs) will accelerate design and optimization in almost all scientific and engineering applications. Neural networks have been receiving ever-increasing attention and demonstrated remarkable success in computational modeling of PDEs, however; their prediction accuracy is not at the level of full deployment. In this work, we utilize the transformer architecture, the backbone of numerous state-of-the-art AI models, to learn the dynamics of physical systems as the mixing of spatial patterns learned by a convolutional autoencoder. Moreover, we incorporate the idea of multi-scale hierarchical time-stepping to increase the prediction speed and decrease accumulated error over time. Our model achieves similar or better results in predicting the time-evolution of Navier-Stokes equations compared to the powerful Fourier Neural Operator (FNO) and two transformer-based neural operators OFormer and Galerkin Transformer.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 34 of its 59 references that are in the map. First posted on arXiv 3 Nov 2023.

Builds on

34 of 59 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020End-to-End Object Detection with Transformerscited by 7,384
  4. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,806
  5. 2019DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operatorscited by 1,965
  6. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  7. 2020Reformer: The Efficient Transformercited by 1,658
  8. 2020Learning to Simulate Complex Physics with Graph Networkscited by 867
  9. 2020Learning Mesh-Based Simulation with Graph Networkscited by 840
  10. 2020Neural Operator: Graph Kernel Network for Partial Differential Equationscited by 722
  11. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  12. 2022Scientific Machine Learning through Physics-Informed Neural Networks: Where we are and What's nextcited by 603
22 more
  1. 2021Physics-informed neural networks (PINNs) for fluid mechanics: A reviewcited by 470
  2. 2019Machine Learning for Fluid Mechanicscited by 380
  3. 2018Efficient Attention: Attention with Linear Complexitiescited by 366
  4. 2021Choose a Transformer: Fourier or Galerkincited by 333
  5. 2023GNOT: A General Neural Operator Transformer for Operator Learningcited by 321
  6. 2021U-FNO -- An enhanced Fourier neural operator-based deep-learning model for multiphase flowcited by 292
  7. 2022Transformer for Partial Differential Equations' Operator Learningcited by 270
  8. 2021Multiwavelet-based Operator Learning for Differential Equationscited by 229
  9. 2019Towards Physics-informed Deep Learning for Turbulent Flow Predictioncited by 203
  10. 2022Towards Multi-spatiotemporal-scale Generalized PDE Modelingcited by 190
  11. 2020PhyGeoNet: Physics-Informed Geometry-Adaptive Convolutional Neural Networks for Solving Parameterized Steady-State PDEs on Irregular Domaincited by 173
  12. 2019Prediction of Aerodynamic Flow Fields Using Convolutional Neural Networkscited by 144
  13. 2020Combining Differentiable PDE Solvers and Graph Neural Networks for Fluid Flow Predictioncited by 137
  14. 2023Scalable Transformer for PDE Surrogate Modelingcited by 123
  15. 2020Transformers for Modeling Physical Systemscited by 105
  16. 2018Latent-space Physics: Towards Learning the Temporal Evolution of Fluid Flowcited by 97
  17. 2022Predicting Physics in Mesh-reduced Space with Temporal Attentioncited by 82
  18. 2023β-Variational autoencoders and transformers for reduced-order modelling of fluid flowscited by 76
  19. 2023Eagle: Large-Scale Learning of Turbulent Fluid Dynamics with Mesh Transformerscited by 54
  20. 2020Hierarchical Deep Learning of Multiscale Differential Equation Time-Stepperscited by 37
  21. 2021Deep Learning for Reduced Order Modelling and Efficient Temporal Evolution of Fluid Simulationscited by 36
  22. 2022Transformer Meets Boundary Value Inverse Problemscited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 3; 2026: 4.

14 papers.

Sort:

  1. 2024Physics-Informed Geometry-Aware Neural Operatorcited by 18
  2. 2024Predicting Change, Not States: An Alternate Framework for Neural PDE Surrogatescited by 8
  3. 2025AMR-Transformer: Enabling Efficient Long-range Interaction for Complex Neural Fluid Simulationcited by 7
  4. 2024PICL: Physics Informed Contrastive Learning for Partial Differential Equationscited by 5
  5. 2024Latent Neural PDE Solver: a reduced-order modelling framework for partial differential equationscited by 5
  6. 2024Pretraining a Neural Operator in Lower Dimensionscited by 5
  7. 2025GALDS: A Graph-Autoencoder-based Latent Dynamics Surrogate model to predict neurite material transportcited by 2
  8. 2026A Physics-Informed Fourier-Wavelet Transformer for Multiscale Computational Fluid Dynamics Surrogate Modelingcited by 2
  9. 2024Inpainting Computational Fluid Dynamics with Deep Learningcited by 1
  10. 2024Flexible SE(2) graph neural networks with applications to PDE surrogatescited by 1
  11. 2025Enhanced accuracy through ensembling of randomly initialized auto-regressive models for time-dependent PDEscited by 0
  12. 2026FlowForge: A Staged Local Rollout Engine for Flow-Field Predictioncited by 0
2 more
  1. 2026An Exploratory Study into using Machine-Learning for Fast Step-by-step Emulation of Numerical Mechanical Thrombectomy Simulations for Ischemic Strokecited by 0
  2. 2026Hypernetwork-Parameterized Spatially Adaptive Neural Operators for PDE Learningcited by 0