# Learning Physical Simulation with Message Passing Transformer

arXiv: https://arxiv.org/abs/2406.06060 · HTML: https://mlsanity.com/paper/2406.06060/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 45 references that are in the map.
- First posted on arXiv 10 Jun 2024.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Fast Graph Representation Learning with PyTorch Geometric](https://mlsanity.com/paper/1903.02428/) (2019)
- [Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering](https://mlsanity.com/paper/1606.09375/) (2016)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [Relational inductive biases, deep learning, and graph networks](https://mlsanity.com/paper/1806.01261/) (2018)
- [Spectral Networks and Locally Connected Networks on Graphs](https://mlsanity.com/paper/1312.6203/) (2013)
- [Learning to Simulate Complex Physics with Graph Networks](https://mlsanity.com/paper/2002.09405/) (2020)
- [Learning Mesh-Based Simulation with Graph Networks](https://mlsanity.com/paper/2010.03409/) (2020)
- [Scientific Machine Learning through Physics-Informed Neural Networks: Where we are and What's next](https://mlsanity.com/paper/2201.05624/) (2022)
- [FastGCN: Fast Learning with Graph Convolutional Networks via Importance Sampling](https://mlsanity.com/paper/1801.10247/) (2018)
- [Graph Transformer Networks](https://mlsanity.com/paper/1911.06455/) (2019)
- [Graph U-Nets](https://mlsanity.com/paper/1905.05178/) (2019)
- [Graph networks as learnable physics engines for inference and control](https://mlsanity.com/paper/1806.01242/) (2018)
- [Multiwavelet-based Operator Learning for Differential Equations](https://mlsanity.com/paper/2109.13459/) (2021)
- [Prediction of Aerodynamic Flow Fields Using Convolutional Neural Networks](https://mlsanity.com/paper/1905.13166/) (2019)
- [Combining Differentiable PDE Solvers and Graph Neural Networks for Fluid Flow Prediction](https://mlsanity.com/paper/2007.04439/) (2020)
- [Accelerating Eulerian Fluid Simulation With Convolutional Networks](https://mlsanity.com/paper/1607.03597/) (2016)
- [Predicting Physics in Mesh-reduced Space with Temporal Attention](https://mlsanity.com/paper/2201.09113/) (2022)
- [Efficient Learning of Mesh-Based Physical Simulation with BSMS-GNN](https://mlsanity.com/paper/2210.02573/) (2022)
- [Learning rigid dynamics with face interaction graph networks](https://mlsanity.com/paper/2212.03574/) (2022)
- [Simulating Continuum Mechanics with Multi-Scale Graph Neural Networks](https://mlsanity.com/paper/2106.04900/) (2021)
- [Constraint-based graph network simulator](https://mlsanity.com/paper/2112.09161/) (2021)
- [Application of Convolutional Neural Network to Predict Airfoil Lift Coefficient](https://mlsanity.com/paper/1712.10082/) (2017)
- [Poisson CNN: Convolutional neural networks for the solution of the Poisson equation on a Cartesian mesh](https://mlsanity.com/paper/1910.08613/) (2019)

## Built on by (most cited first, 3 of 3)

- [Pay Attention to Attention Distribution: A New Local Lipschitz Bound for Transformers](https://mlsanity.com/paper/2507.07814/) (2025)
- [AMR-Transformer: Enabling Efficient Long-range Interaction for Complex Neural Fluid Simulation](https://mlsanity.com/paper/2503.10257/) (2025)
- [Beyond Scanpaths: Graph-Based Gaze Simulation in Dynamic Scenes](https://mlsanity.com/paper/2603.28319/) (2026)

## Abstract (arXiv)

Machine learning methods for physical simulation have achieved significant success in recent years. We propose a new universal architecture based on Graph Neural Network, the Message Passing Transformer, which incorporates a Message Passing framework, employs an Encoder-Processor-Decoder structure, and applies Graph Fourier Loss as loss function for model optimization. To take advantage of the past message passing state information, we propose Hadamard-Product Attention to update the node attribute in the Processor, Hadamard-Product Attention is a variant of Dot-Product Attention that focuses on more fine-grained semantics and emphasizes on assigning attention weights over each feature dimension rather than each position in the sequence relative to others. We further introduce Graph Fourier Loss (GFL) to balance high-energy and low-energy components. To improve time performance, we precompute the graph's Laplacian eigenvectors before the training process. Our architecture achieves significant accuracy improvements in long-term rollouts for both Lagrangian and Eulerian dynamical systems over current methods.