# Towards Stability of Autoregressive Neural Operators

arXiv: https://arxiv.org/abs/2306.10619 · HTML: https://mlsanity.com/paper/2306.10619/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 81 references that are in the map.
- First posted on arXiv 18 Jun 2023.

## Builds on (most cited first, 30 of 43)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [Alias-Free Generative Adversarial Networks](https://mlsanity.com/paper/2106.12423/) (2021)
- [Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/1702.03118/) (2017)
- [FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators](https://mlsanity.com/paper/2202.11214/) (2022)
- [Language Modeling with Gated Convolutional Networks](https://mlsanity.com/paper/1612.08083/) (2016)
- [Large Batch Optimization for Deep Learning: Training BERT in 76 minutes](https://mlsanity.com/paper/1904.00962/) (2019)
- [Machine learning accelerated computational fluid dynamics](https://mlsanity.com/paper/2102.01010/) (2021)
- [Spherical CNNs](https://mlsanity.com/paper/1801.10130/) (2018)
- [Making Convolutional Networks Shift-Invariant Again](https://mlsanity.com/paper/1904.11486/) (2019)
- [Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness](https://mlsanity.com/paper/2006.10108/) (2020)
- [Dynamic Convolution: Attention over Convolution Kernels](https://mlsanity.com/paper/1912.03458/) (2019)
- [U-FNO -- An enhanced Fourier neural operator-based deep-learning model for multiphase flow](https://mlsanity.com/paper/2109.03697/) (2021)
- [GraphCast: Learning skillful medium-range global weather forecasting](https://mlsanity.com/paper/2212.12794/) (2022)
- [Factorized Fourier Neural Operators](https://mlsanity.com/paper/2111.13802/) (2021)
- [Learning SO(3) Equivariant Representations with Spherical CNNs](https://mlsanity.com/paper/1711.06721/) (2017)
- [Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models](https://mlsanity.com/paper/2208.06677/) (2022)
- [Pangu-Weather: A 3D High-Resolution Model for Fast and Accurate Global Weather Forecast](https://mlsanity.com/paper/2211.02556/) (2022)

## Built on by (most cited first, 30 of 59)

- [The Well: a Large-Scale Collection of Diverse Physics Simulations for Machine Learning](https://mlsanity.com/paper/2412.00568/) (2024)
- [Walrus: A Cross-Domain Foundation Model for Continuum Dynamics](https://mlsanity.com/paper/2511.15684/) (2025)
- [LUCIE: A Lightweight Uncoupled ClImate Emulator with long-term stability and physical consistency for O(1000)-member ensembles](https://mlsanity.com/paper/2405.16297/) (2024)
- [Generative Latent Neural PDE Solver using Flow Matching](https://mlsanity.com/paper/2503.22600/) (2025)
- [DISCO: learning to DISCover an evolution Operator for multi-physics-agnostic prediction](https://mlsanity.com/paper/2504.19496/) (2025)
- [Lost in Latent Space: An Empirical Study of Latent Diffusion Models for Physics Emulation](https://mlsanity.com/paper/2507.02608/) (2025)
- [BubbleML: A Multi-Physics Dataset and Benchmarks for Machine Learning](https://mlsanity.com/paper/2307.14623/) (2023)
- [Mixtures of Neural Operators Reduce Active Complexity in Operator Learning](https://mlsanity.com/paper/2404.09101/) (2024)
- [Recurrent Neural Operators: Stable Long-Term PDE Prediction](https://mlsanity.com/paper/2505.20721/) (2025)
- [Neural Green's Operators for Parametric Partial Differential Equations](https://mlsanity.com/paper/2406.01857/) (2024)
- [Exploring the design space of deep-learning-based weather forecasting systems](https://mlsanity.com/paper/2410.07472/) (2024)
- [Calibrated Physics-Informed Uncertainty Quantification](https://mlsanity.com/paper/2502.04406/) (2025)
- [Quantum-Informed Machine Learning for Predicting Spatiotemporal Chaos with Practical Quantum Advantage](https://mlsanity.com/paper/2507.19861/) (2025)
- [Neural Emulator Superiority: When Machine Learning for PDEs Surpasses its Training Data](https://mlsanity.com/paper/2510.23111/) (2025)
- [Bubbleformer: Forecasting Boiling with Transformers](https://mlsanity.com/paper/2507.21244/) (2025)
- [Physics-Informed Time-Integrated DeepONet: Temporal Tangent Space Operator Learning for High-Accuracy Inference](https://mlsanity.com/paper/2508.05190/) (2025)
- [(Sparse) Attention to the Details: Preserving Spectral Fidelity in ML-based Weather Forecasting Models](https://mlsanity.com/paper/2604.16429/) (2026)
- [Symplectic Neural Flows for Modeling and Discovery](https://mlsanity.com/paper/2412.16787/) (2024)
- [Uni-Flow: a unified autoregressive-diffusion model for complex multiscale flows](https://mlsanity.com/paper/2602.15592/) (2026)
- [Can AI Weather Models Predict Beyond Two Weeks? A Quantitative Benchmark and Analysis of Long Rollouts](https://mlsanity.com/paper/2605.30184/) (2026)
- [INC: An Indirect Neural Corrector for Auto-Regressive Hybrid PDE Solvers](https://mlsanity.com/paper/2511.12764/) (2025)
- [Controlling Transient Amplification Improves Long-horizon Rollouts](https://mlsanity.com/paper/2605.08856/) (2026)
- [MoWE : A Mixture of Weather Experts](https://mlsanity.com/paper/2509.09052/) (2025)
- [Learning Physical Operators using Neural Operators](https://mlsanity.com/paper/2602.23113/) (2026)
- [Flowers: A Warp Drive for Neural PDE Solvers](https://mlsanity.com/paper/2603.04430/) (2026)
- [StFT: Spatio-temporal Fourier Transformer for Long-term Dynamics Prediction](https://mlsanity.com/paper/2503.11899/) (2025)
- [Exploring Design Choices for Autoregressive Deep Learning Climate Models](https://mlsanity.com/paper/2505.02506/) (2025)
- [Neural-HSS: Hierarchical Semi-Separable Neural PDE Solver](https://mlsanity.com/paper/2602.18248/) (2026)
- [Physics-informed neural operator for parametric phase-field modelling of interfacial degradation and microstructural evolution](https://mlsanity.com/paper/2603.09693/) (2026)
- [Factorized Neural Implicit DMD for Parametric Dynamics](https://mlsanity.com/paper/2603.10995/) (2026)

## Abstract (arXiv)

Neural operators have proven to be a promising approach for modeling spatiotemporal systems in the physical sciences. However, training these models for large systems can be quite challenging as they incur significant computational and memory expense -- these systems are often forced to rely on autoregressive time-stepping of the neural network to predict future temporal states. While this is effective in managing costs, it can lead to uncontrolled error growth over time and eventual instability. We analyze the sources of this autoregressive error growth using prototypical neural operator models for physical systems and explore ways to mitigate it. We introduce architectural and application-specific improvements that allow for careful control of instability-inducing operations within these models without inflating the compute/memory expense. We present results on several scientific systems that include Navier-Stokes fluid flow, rotating shallow water, and a high-resolution global weather forecasting system. We demonstrate that applying our design principles to neural operators leads to significantly lower errors for long-term forecasts as well as longer time horizons without qualitative signs of divergence compared to the original models for these systems. We open-source our hrefhttps://github.com/mikemccabe210/stabilizingneuraloperatorscode for reproducibility.