# Enhanced accuracy through ensembling of randomly initialized auto-regressive models for time-dependent PDEs

arXiv: https://arxiv.org/abs/2507.03863 · HTML: https://mlsanity.com/paper/2507.03863/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 37 references that are in the map.
- First posted on arXiv 5 Jul 2025.

## Builds on (most cited first, 22 of 22)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [When and why PINNs fail to train: A neural tangent kernel perspective](https://mlsanity.com/paper/2007.14527/) (2020)
- [Deep Ensembles: A Loss Landscape Perspective](https://mlsanity.com/paper/1912.02757/) (2019)
- [Transolver: A Fast Transformer Solver for PDEs on General Geometries](https://mlsanity.com/paper/2402.02366/) (2024)
- [The Well: a Large-Scale Collection of Diverse Physics Simulations for Machine Learning](https://mlsanity.com/paper/2412.00568/) (2024)
- [Scalable Transformer for PDE Surrogate Modeling](https://mlsanity.com/paper/2305.17560/) (2023)
- [Transformers for Modeling Physical Systems](https://mlsanity.com/paper/2010.03957/) (2020)
- [Dynamics of Deep Neural Networks and Neural Tangent Hierarchy](https://mlsanity.com/paper/1909.08156/) (2019)
- [Learning two-phase microstructure evolution using neural operators and autoencoder architectures](https://mlsanity.com/paper/2204.07230/) (2022)
- [Towards Stability of Autoregressive Neural Operators](https://mlsanity.com/paper/2306.10619/) (2023)
- [Sequential Deep Operator Networks (S-DeepONet) for Predicting Full-field Solutions Under Time-dependent Loads](https://mlsanity.com/paper/2306.08218/) (2023)
- [Learning stiff chemical kinetics using extended deep neural operators](https://mlsanity.com/paper/2302.12645/) (2023)
- [Physics guided neural networks for modelling of non-linear dynamics](https://mlsanity.com/paper/2205.06858/) (2022)
- [Real-time Inference and Extrapolation via a Diffusion-inspired Temporal Transformer Operator (DiTTO)](https://mlsanity.com/paper/2307.09072/) (2023)
- [Multi-scale Time-stepping of Partial Differential Equations with Transformers](https://mlsanity.com/paper/2311.02225/) (2023)
- [TI-DeepONet: Learnable Time Integration for Stable Long-Term Extrapolation](https://mlsanity.com/paper/2505.17341/) (2025)
- [Learning and discovering multiple solutions using physics-informed neural networks with random initialization and deep ensemble](https://mlsanity.com/paper/2503.06320/) (2025)
- [A foundational neural operator that continuously learns without forgetting](https://mlsanity.com/paper/2310.18885/) (2023)
- [Ensemble and Mixture-of-Experts DeepONets For Operator Learning](https://mlsanity.com/paper/2405.11907/) (2024)

## Abstract (arXiv)

Systems governed by partial differential equations (PDEs) require computationally intensive numerical solvers to predict spatiotemporal field evolution. While machine learning (ML) surrogates offer faster solutions, autoregressive inference with ML models suffer from error accumulation over successive predictions, limiting their long-term accuracy. We propose a deep ensemble framework to address this challenge, where multiple ML surrogate models with random weight initializations are trained in parallel and aggregated during inference. This approach leverages the diversity of model predictions to mitigate error propagation while retaining the autoregressive strategies ability to capture the system's time dependent relations. We validate the framework on three PDE-driven dynamical systems - stress evolution in heterogeneous microstructures, Gray-Scott reaction-diffusion, and planetary-scale shallow water system - demonstrating consistent reduction in error accumulation over time compared to individual models. Critically, the method requires only a few time steps as input, enabling full trajectory predictions with inference times significantly faster than numerical solvers. Our results highlight the robustness of ensemble methods in diverse physical systems and their potential as efficient and accurate alternatives to traditional solvers. The codes for this work are available on GitHub (https://github.com/Graham-Brady-Research-Group/AutoregressiveEnsemble_SpatioTemporal_Evolution).