# Attention mechanism for scalable mesh-based neural surrogates of free-surface fluids

arXiv: https://arxiv.org/abs/2606.23251 · HTML: https://mlsanity.com/paper/2606.23251/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 56 references that are in the map.
- First posted on arXiv 22 Jun 2026.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Relational inductive biases, deep learning, and graph networks](https://mlsanity.com/paper/1806.01261/) (2018)
- [Learning to Simulate Complex Physics with Graph Networks](https://mlsanity.com/paper/2002.09405/) (2020)
- [Efficient Attention: Attention with Linear Complexities](https://mlsanity.com/paper/1812.01243/) (2018)
- [Choose a Transformer: Fourier or Galerkin](https://mlsanity.com/paper/2105.14995/) (2021)
- [Transolver: A Fast Transformer Solver for PDEs on General Geometries](https://mlsanity.com/paper/2402.02366/) (2024)
- [Geometry-Informed Neural Operator for Large-Scale 3D PDEs](https://mlsanity.com/paper/2309.00583/) (2023)
- [Transformer for Partial Differential Equations' Operator Learning](https://mlsanity.com/paper/2205.13671/) (2022)
- [A Survey on Oversmoothing in Graph Neural Networks](https://mlsanity.com/paper/2303.10993/) (2023)
- [Geometry Aware Operator Transformer as an Efficient and Accurate Neural Surrogate for PDEs on Arbitrary Domains](https://mlsanity.com/paper/2505.18781/) (2025)
- [Towards Stability of Autoregressive Neural Operators](https://mlsanity.com/paper/2306.10619/) (2023)
- [A Comprehensive Study on Large-Scale Graph Training: Benchmarking and Rethinking](https://mlsanity.com/paper/2210.07494/) (2022)
- [AB-UPT: Scaling Neural CFD Surrogates for High-Fidelity Automotive Aerodynamics Simulations via Anchored-Branched Universal Physics Transformers](https://mlsanity.com/paper/2502.09692/) (2025)
- [Graph Neural Networks Accelerated Molecular Dynamics](https://mlsanity.com/paper/2112.03383/) (2021)
- [GeoTransolver: Learning Physics on Irregular Domains Using Multi-scale Geometry Aware Physics Attention Transformer](https://mlsanity.com/paper/2512.20399/) (2025)
- [Transolver-3: Scaling Up Transformer Solvers to Industrial-Scale Geometries](https://mlsanity.com/paper/2602.04940/) (2026)
- [Graph Neural Network-based surrogate model for granular flows](https://mlsanity.com/paper/2305.05218/) (2023)
- [NeuralDEM -- Real-time Simulation of Industrial Particulate Flows](https://mlsanity.com/paper/2411.09678/) (2024)
- [MeshGraphNet-Transformer: Scalable Mesh-based Learned Simulation for Solid Mechanics](https://mlsanity.com/paper/2601.23177/) (2026)
- [Handling geometrical variability in nonlinear reduced order modeling through Continuous Geometry-Aware DL-ROMs](https://mlsanity.com/paper/2411.05486/) (2024)
- [On the under-reaching phenomenon in message-passing neural PDE solvers: revisiting the CFL condition](https://mlsanity.com/paper/2507.08861/) (2025)
- [Improving Long-term Autoregressive Spatiotemporal Predictions: A Proof of Concept with Fluid Dynamics](https://mlsanity.com/paper/2508.18565/) (2025)
- [FluidFormer: Transformer with Continuous Convolution for Particle-based Fluid Simulation](https://mlsanity.com/paper/2508.01537/) (2025)
- [RheOFormer: A generative transformer model for simulation of complex fluids and flows](https://mlsanity.com/paper/2510.01365/) (2025)
- [ShapeGen3DCP: A Deep Learning Framework for Layer Shape Prediction in 3D Concrete Printing](https://mlsanity.com/paper/2510.02009/) (2025)

## Abstract (arXiv)

High-fidelity simulations of free-surface flows using Lagrangian methods such as the Particle Finite Element Method (PFEM) are computationally demanding due to continuous domain updates and repeated solution of the governing equations. This challenge is further amplified by non-Newtonian rheologies, where material nonlinearities increase computational cost. These limitations motivate the development of efficient surrogate models to approximate PFEM dynamics at reduced cost. While data-driven deep learning approaches are promising, a key challenge is designing models that operate on arbitrary and evolving geometries. We propose a self-attention-based neural surrogate for PFEM simulations of free-surface flows. The architecture leverages attention mechanisms to model node interactions and capture complex spatial dependencies, while preserving the PFEM mesh discretization. This provides a geometric and topological framework for remeshing and node redistribution, maintaining high-quality spatial discretization during rollouts, improving long-term stability, and enabling reconstruction of derived mechanical quantities via standard finite element operators. Two attention formulations are considered: a standard self-attention mechanism and a linear variant that reduces computational cost and improves scalability. The models are evaluated on two- and three-dimensional free-surface flow benchmarks with evolving geometries, varying material parameters, and non-Newtonian fluids. Results show accurate prediction of transient dynamics and final configurations, with significantly improved scalability. The mesh-based formulation also enables direct reconstruction of quantities such as stress fields. Overall, the framework provides an accurate and scalable surrogate strategy for PFEM simulations in engineering-scale applications.