# Multi-GPU Approach for Training of Graph ML Models on large CFD Meshes

arXiv: https://arxiv.org/abs/2307.13592 · HTML: https://mlsanity.com/paper/2307.13592/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 31 references that are in the map.
- First posted on arXiv 25 Jul 2023.

## Builds on (most cited first, 13 of 13)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [Learning Mesh-Based Simulation with Graph Networks](https://mlsanity.com/paper/2010.03409/) (2020)
- [Horovod: fast and easy distributed deep learning in TensorFlow](https://mlsanity.com/paper/1802.05799/) (2018)
- [Deep Learning Methods for Reynolds-Averaged Navier-Stokes Simulations of Airfoil Flows](https://mlsanity.com/paper/1810.08217/) (2018)
- [Combining Differentiable PDE Solvers and Graph Neural Networks for Fluid Flow Prediction](https://mlsanity.com/paper/2007.04439/) (2020)
- [PyTorch-BigGraph: A Large-scale Graph Embedding System](https://mlsanity.com/paper/1903.12287/) (2019)
- [MultiScale MeshGraphNets](https://mlsanity.com/paper/2210.00612/) (2022)
- [A Point-Cloud Deep Learning Framework for Prediction of Fluid Flow Fields on Irregular Geometries](https://mlsanity.com/paper/2010.09469/) (2020)
- [Simulating Continuum Mechanics with Multi-Scale Graph Neural Networks](https://mlsanity.com/paper/2106.04900/) (2021)
- [Towards Fast Simulation of Environmental Fluid Mechanics with Multi-Scale Graph Neural Networks](https://mlsanity.com/paper/2205.02637/) (2022)
- [Advanced Python Performance Monitoring with Score-P](https://mlsanity.com/paper/2010.15444/) (2020)

## Built on by (most cited first, 4 of 4)

- [SEGNO: Generalizing Equivariant Graph Neural Networks with Physical Inductive Biases](https://mlsanity.com/paper/2308.13212/) (2023)
- [Sampling-based Distributed Training with Message Passing Neural Network](https://mlsanity.com/paper/2402.15106/) (2024)
- [Scalable and Consistent Graph Neural Networks for Distributed Mesh-based Data-driven Modeling](https://mlsanity.com/paper/2410.01657/) (2024)
- [A Mesh-Adaptive Hypergraph Neural Network for Unsteady Flow Around Oscillating and Rotating Structures](https://mlsanity.com/paper/2503.22252/) (2025)

## Abstract (arXiv)

Mesh-based numerical solvers are an important part in many design tool chains. However, accurate simulations like computational fluid dynamics are time and resource consuming which is why surrogate models are employed to speed-up the solution process. Machine Learning based surrogate models on the other hand are fast in predicting approximate solutions but often lack accuracy. Thus, the development of the predictor in a predictor-corrector approach is the focus here, where the surrogate model predicts a flow field and the numerical solver corrects it. This paper scales a state-of-the-art surrogate model from the domain of graph-based machine learning to industry-relevant mesh sizes of a numerical flow simulation. The approach partitions and distributes the flow domain to multiple GPUs and provides halo exchange between these partitions during training. The utilized graph neural network operates directly on the numerical mesh and is able to preserve complex geometries as well as all other properties of the mesh. The proposed surrogate model is evaluated with an application on a three dimensional turbomachinery setup and compared to a traditionally trained distributed model. The results show that the traditional approach produces superior predictions and outperforms the proposed surrogate model. Possible explanations, improvements and future directions are outlined.