# Weak baselines and reporting biases lead to overoptimism in machine learning for fluid-related partial differential equations

arXiv: https://arxiv.org/abs/2407.07218 · HTML: https://mlsanity.com/paper/2407.07218/

## Key facts

- Cited by 91 papers in the map (data through 14 Oct 2026).
- Builds on 71 of its 190 references that are in the map.
- First posted on arXiv 9 Jul 2024.

## Builds on (most cited first, 30 of 71)

- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [Learning to Simulate Complex Physics with Graph Networks](https://mlsanity.com/paper/2002.09405/) (2020)
- [Learning Mesh-Based Simulation with Graph Networks](https://mlsanity.com/paper/2010.03409/) (2020)
- [Characterizing possible failure modes in physics-informed neural networks](https://mlsanity.com/paper/2109.01050/) (2021)
- [When and why PINNs fail to train: A neural tangent kernel perspective](https://mlsanity.com/paper/2007.14527/) (2020)
- [Scientific Machine Learning through Physics-Informed Neural Networks: Where we are and What's next](https://mlsanity.com/paper/2201.05624/) (2022)
- [Learning the solution operator of parametric partial differential equations with physics-informed DeepOnets](https://mlsanity.com/paper/2103.10974/) (2021)
- [Physics-Informed Neural Operator for Learning Partial Differential Equations](https://mlsanity.com/paper/2111.03794/) (2021)
- [Machine learning accelerated computational fluid dynamics](https://mlsanity.com/paper/2102.01010/) (2021)
- [Fourier Neural Operator with Learned Deformations for PDEs on General Geometries](https://mlsanity.com/paper/2207.05209/) (2022)
- [Neural Operator: Learning Maps Between Function Spaces](https://mlsanity.com/paper/2108.08481/) (2021)
- [Message Passing Neural PDE Solvers](https://mlsanity.com/paper/2202.03376/) (2022)
- [PDE-Refiner: Achieving Accurate Long Rollouts with Neural PDE Solvers](https://mlsanity.com/paper/2308.05732/) (2023)
- [Solver-in-the-Loop: Learning from Differentiable Physics to Interact with Iterative PDE-Solvers](https://mlsanity.com/paper/2007.00016/) (2020)
- [Towards Multi-spatiotemporal-scale Generalized PDE Modeling](https://mlsanity.com/paper/2209.15616/) (2022)
- [Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans](https://mlsanity.com/paper/2008.06388/) (2020)
- [Enhancing Computational Fluid Dynamics with Machine Learning](https://mlsanity.com/paper/2110.02085/) (2021)
- [Backpropagation Algorithms and Reservoir Computing in Recurrent Neural Networks for the Forecasting of Complex Spatiotemporal Dynamics](https://mlsanity.com/paper/1910.05266/) (2019)
- [Accelerating Eulerian Fluid Simulation With Convolutional Networks](https://mlsanity.com/paper/1607.03597/) (2016)
- [Deep Fluids: A Generative Network for Parameterized Fluid Simulations](https://mlsanity.com/paper/1806.02071/) (2018)
- [Learned Coarse Models for Efficient Turbulence Simulation](https://mlsanity.com/paper/2112.15275/) (2021)
- [Long-time integration of parametric evolution equations with physics-informed DeepONets](https://mlsanity.com/paper/2106.05384/) (2021)
- [Improved architectures and training algorithms for deep operator networks](https://mlsanity.com/paper/2110.01654/) (2021)
- [Predicting Physics in Mesh-reduced Space with Temporal Attention](https://mlsanity.com/paper/2201.09113/) (2022)
- [Local Extreme Learning Machines and Domain Decomposition for Solving Linear and Nonlinear Partial Differential Equations](https://mlsanity.com/paper/2012.02895/) (2020)
- [Learning to Optimize Multigrid PDE Solvers](https://mlsanity.com/paper/1902.10248/) (2019)
- [Learning Neural PDE Solvers with Convergence Guarantees](https://mlsanity.com/paper/1906.01200/) (2019)
- [Physics-based Deep Learning](https://mlsanity.com/paper/2109.05237/) (2021)
- [Can Physics-Informed Neural Networks beat the Finite Element Method?](https://mlsanity.com/paper/2302.04107/) (2023)

## Built on by (most cited first, 30 of 91)

- [On the Benefits of Memory for Modeling Time-Dependent PDEs](https://mlsanity.com/paper/2409.02313/) (2024)
- [Can AI weather models predict out-of-distribution gray swan tropical cyclones?](https://mlsanity.com/paper/2410.14932/) (2024)
- [Text2PDE: Latent Diffusion Models for Accessible Physics Simulation](https://mlsanity.com/paper/2410.01153/) (2024)
- [Erwin: A Tree-based Hierarchical Transformer for Large-scale Physical Systems](https://mlsanity.com/paper/2502.17019/) (2025)
- [Sparse identification of nonlinear dynamics and Koopman operators with Shallow Recurrent Decoder Networks](https://mlsanity.com/paper/2501.13329/) (2025)
- [Physics-informed kernel learning](https://mlsanity.com/paper/2409.13786/) (2024)
- [Evolutionary Optimization of Physics-Informed Neural Networks: Evo-PINN Frontiers and Opportunities](https://mlsanity.com/paper/2501.06572/) (2025)
- [Neural equilibria for long-term prediction of nonlinear conservation laws](https://mlsanity.com/paper/2501.06933/) (2025)
- [Enhancing Fourier Neural Operators with Local Spatial Features](https://mlsanity.com/paper/2503.17797/) (2025)
- [Predicting Change, Not States: An Alternate Framework for Neural PDE Surrogates](https://mlsanity.com/paper/2412.13074/) (2024)
- [Solving Sparse Finite Element Problems on Neuromorphic Hardware](https://mlsanity.com/paper/2501.10526/) (2025)
- [The wall confronting large language models](https://mlsanity.com/paper/2507.19703/) (2025)
- [Accelerating scientific discovery with the common task framework](https://mlsanity.com/paper/2511.04001/) (2025)
- [Performance of Neural and Polynomial Operator Surrogates](https://mlsanity.com/paper/2604.00689/) (2026)
- [Neural Emulator Superiority: When Machine Learning for PDEs Surpasses its Training Data](https://mlsanity.com/paper/2510.23111/) (2025)
- [EddyFormer: Accelerated Neural Simulations of Three-Dimensional Turbulence at Scale](https://mlsanity.com/paper/2510.24173/) (2025)
- [When is a System Discoverable from Data? Discovery Requires Chaos](https://mlsanity.com/paper/2511.08860/) (2025)
- [The Seismic Wavefield Common Task Framework](https://mlsanity.com/paper/2512.19927/) (2025)
- [What You See is Not What You Get: Neural Partial Differential Equations and The Illusion of Learning](https://mlsanity.com/paper/2411.15101/) (2024)
- [FD-Bench: A Modular and Fair Benchmark for Data-driven Fluid Simulation](https://mlsanity.com/paper/2505.20349/) (2025)
- [Scale-PINN: Learning Efficient Physics-Informed Neural Networks Through Sequential Correction](https://mlsanity.com/paper/2602.19475/) (2026)
- [CTF4Nuclear: Common Task Framework for Nuclear Fission and Fusion Models](https://mlsanity.com/paper/2605.15549/) (2026)
- [Symplectic Neural Flows for Modeling and Discovery](https://mlsanity.com/paper/2412.16787/) (2024)
- [Verification and Validation for Trustworthy Scientific Machine Learning](https://mlsanity.com/paper/2502.15496/) (2025)
- [Neural Quantum Propagators for Driven-Dissipative Quantum Dynamics](https://mlsanity.com/paper/2410.16091/) (2024)
- [Solving Differential Equations with Constrained Learning](https://mlsanity.com/paper/2410.22796/) (2024)
- [Beyond Accuracy: EcoL2 Metric for Sustainable Neural PDE Solvers](https://mlsanity.com/paper/2505.12556/) (2025)
- [Neural Interpretable PDEs: Harmonizing Fourier Insights with Attention for Scalable and Interpretable Physics Discovery](https://mlsanity.com/paper/2505.23106/) (2025)
- [ReBaNO: Reduced Basis Neural Operator Mitigating Generalization Gaps and Achieving Discretization Invariance](https://mlsanity.com/paper/2509.09611/) (2025)
- [Examining the robustness of Physics-Informed Neural Networks to noise for Inverse Problems](https://mlsanity.com/paper/2509.20191/) (2025)

## Abstract (arXiv)

One of the most promising applications of machine learning (ML) in computational physics is to accelerate the solution of partial differential equations (PDEs). The key objective of ML-based PDE solvers is to output a sufficiently accurate solution faster than standard numerical methods, which are used as a baseline comparison. We first perform a systematic review of the ML-for-PDE solving literature. Of articles that use ML to solve a fluid-related PDE and claim to outperform a standard numerical method, we determine that 79% (60/76) compare to a weak baseline. Second, we find evidence that reporting biases, especially outcome reporting bias and publication bias, are widespread. We conclude that ML-for-PDE solving research is overoptimistic: weak baselines lead to overly positive results, while reporting biases lead to underreporting of negative results. To a large extent, these issues appear to be caused by factors similar to those of past reproducibility crises: researcher degrees of freedom and a bias towards positive results. We call for bottom-up cultural changes to minimize biased reporting as well as top-down structural reforms intended to reduce perverse incentives for doing so.