# Differentiability in Unrolled Training of Neural Physics Simulators on Transient Dynamics

arXiv: https://arxiv.org/abs/2402.12971 · HTML: https://mlsanity.com/paper/2402.12971/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 72 references that are in the map.
- First posted on arXiv 20 Feb 2024.

## Builds on (most cited first, 13 of 13)

- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Attention U-Net: Learning Where to Look for the Pancreas](https://mlsanity.com/paper/1804.03999/) (2018)
- [Learning Mesh-Based Simulation with Graph Networks](https://mlsanity.com/paper/2010.03409/) (2020)
- [Transformer for Partial Differential Equations' Operator Learning](https://mlsanity.com/paper/2205.13671/) (2022)
- [Modern Koopman Theory for Dynamical Systems](https://mlsanity.com/paper/2102.12086/) (2021)
- [GraphCast: Learning skillful medium-range global weather forecasting](https://mlsanity.com/paper/2212.12794/) (2022)
- [DiffTaichi: Differentiable Programming for Physical Simulation](https://mlsanity.com/paper/1910.00935/) (2019)
- [PDE-Refiner: Achieving Accurate Long Rollouts with Neural PDE Solvers](https://mlsanity.com/paper/2308.05732/) (2023)
- [A Fine-Grained Analysis on Distribution Shift](https://mlsanity.com/paper/2110.11328/) (2021)
- [Learned Coarse Models for Efficient Turbulence Simulation](https://mlsanity.com/paper/2112.15275/) (2021)
- [Benchmarking Autoregressive Conditional Diffusion Models for Turbulent Flow Simulation](https://mlsanity.com/paper/2309.01745/) (2023)
- [From Zero to Turbulence: Generative Modeling for 3D Flow Simulation](https://mlsanity.com/paper/2306.01776/) (2023)
- [Koopman Neural Forecaster for Time Series with Temporal Distribution Shifts](https://mlsanity.com/paper/2210.03675/) (2022)

## Built on by (most cited first, 27 of 27)

- [Generative Latent Neural PDE Solver using Flow Matching](https://mlsanity.com/paper/2503.22600/) (2025)
- [Thermalizer: Stable autoregressive neural emulation of spatiotemporal chaos](https://mlsanity.com/paper/2503.18731/) (2025)
- [Hierarchical Implicit Neural Emulators](https://mlsanity.com/paper/2506.04528/) (2025)
- [Neural Emulator Superiority: When Machine Learning for PDEs Surpasses its Training Data](https://mlsanity.com/paper/2510.23111/) (2025)
- [A Neural Material Point Method for Particle-based Emulation](https://mlsanity.com/paper/2408.15753/) (2024)
- [What You See is Not What You Get: Neural Partial Differential Equations and The Illusion of Learning](https://mlsanity.com/paper/2411.15101/) (2024)
- [Model-Agnostic AI Framework with Explicit Time Integration for Long-Term Fluid Dynamics Prediction](https://mlsanity.com/paper/2412.05657/) (2024)
- [INC: An Indirect Neural Corrector for Auto-Regressive Hybrid PDE Solvers](https://mlsanity.com/paper/2511.12764/) (2025)
- [Controlling Transient Amplification Improves Long-horizon Rollouts](https://mlsanity.com/paper/2605.08856/) (2026)
- [Differentiable Autoencoding Neural Operator for Interpretable and Integrable Latent Space Modeling](https://mlsanity.com/paper/2510.00233/) (2025)
- [Hypothesis-driven construction of mesoscopic dynamics](https://mlsanity.com/paper/2605.16211/) (2026)
- [Implicit Neural Differential Model for Spatiotemporal Dynamics](https://mlsanity.com/paper/2504.02260/) (2025)
- [A Multimodal Vision Transformer-based Modeling Framework for Prediction of Fluid Flows in Energy Systems](https://mlsanity.com/paper/2604.02483/) (2026)
- [HERO: History-Enriched Rollout Training for Long-Horizon Autoregressive Neural Operators](https://mlsanity.com/paper/2607.29135/) (2026)
- [Hamiltonian Neural PDE Solvers through Functional Approximation](https://mlsanity.com/paper/2505.13275/) (2025)
- [Zero-Shot Function Encoder-Based Differentiable Predictive Control](https://mlsanity.com/paper/2511.05757/) (2025)
- [Learning collision operators from plasma phase space data using differentiable simulators](https://mlsanity.com/paper/2601.10885/) (2026)
- [Reduced-Order Surrogates for Forced Flexible Mesh Coastal-Ocean Models](https://mlsanity.com/paper/2602.05416/) (2026)
- [Integrating Mechanistic and Data-Driven Models for Neurological Disorders through Differentiable Programming](https://mlsanity.com/paper/2606.06094/) (2026)
- [Mosaic: A Benchmark Suite for Differentiable Physics Solvers](https://mlsanity.com/paper/2606.27895/) (2026)
- [Guarantees by Construction for Learned Finite Volume Schemes on Steady Supersonic Flow](https://mlsanity.com/paper/2607.20171/) (2026)
- [Large Distant Gradients Need Not Be Reliable: reliability-weighted credit assignment for long-horizon autoregressive forecasting](https://mlsanity.com/paper/2609.12890/) (2026)
- [Mixed-Integer Nonlinear Differentiable Predictive Control for Underground Pumped Hydro Energy Storage Systems](https://mlsanity.com/paper/2609.17964/) (2026)
- [Staying on the Attractor: Supervising Neural Surrogates of 3D Turbulence Where They Leave It](https://mlsanity.com/paper/2609.32864/) (2026)
- [CasEm: A Cascade Architecture for Long-Horizon Neural Emulation](https://mlsanity.com/paper/2609.34246/) (2026)
- [Green-Routed Neural Operators: Physics Determines Where the Network Reads](https://mlsanity.com/paper/2610.05337/) (2026)
- [Learning When to Refine: Long-Horizon Reinforcement Learning for Budgeted Neural-Operator PDE Solvers](https://mlsanity.com/paper/2610.06883/) (2026)

## Abstract (arXiv)

Unrolling training trajectories over time strongly influences the inference accuracy of neural network-augmented physics simulators. We analyze this in three variants of training neural time-steppers. In addition to one-step setups and fully differentiable unrolling, we include a third, less widely used variant: unrolling without temporal gradients. Comparing networks trained with these three modalities disentangles the two dominant effects of unrolling, training distribution shift and long-term gradients. We present detailed study across physical systems, network sizes and architectures, training setups, and test scenarios. It also encompasses two simulation modes: In prediction setups, we rely solely on neural networks to compute a trajectory. In contrast, correction setups include a numerical solver that is supported by a neural network. Spanning these variations, our study provides the empirical basis for our main findings: Non-differentiable but unrolled training with a numerical solver in a correction setup can yield substantial improvements over a fully differentiable prediction setup not utilizing this solver. The accuracy of models trained in a fully differentiable setup differs compared to their non-differentiable counterparts. Differentiable ones perform best in a comparison among correction networks as well as among prediction setups. For both, the accuracy of non-differentiable unrolling comes close. Furthermore, we show that these behaviors are invariant to the physical system, the network architecture and size, and the numerical scheme. These results motivate integrating non-differentiable numerical simulators into training setups even if full differentiability is unavailable. We show the convergence rate of common architectures to be low compared to numerical algorithms. This motivates correction setups combining neural and numerical parts which utilize benefits of both.