# Stabilizing Machine Learning Prediction of Dynamics: Noise and Noise-inspired Regularization

arXiv: https://arxiv.org/abs/2211.05262 · HTML: https://mlsanity.com/paper/2211.05262/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 46 references that are in the map.
- First posted on arXiv 9 Nov 2022.

## Builds on (most cited first, 14 of 14)

- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators](https://mlsanity.com/paper/2202.11214/) (2022)
- [WeatherBench: A benchmark dataset for data-driven weather forecasting](https://mlsanity.com/paper/2002.00469/) (2020)
- [Professor Forcing: A New Algorithm for Training Recurrent Networks](https://mlsanity.com/paper/1610.09038/) (2016)
- [Deep learning to represent sub-grid processes in climate models](https://mlsanity.com/paper/1806.04731/) (2018)
- [Data-Driven Forecasting of High-Dimensional Chaotic Systems with Long Short-Term Memory Networks](https://mlsanity.com/paper/1802.07486/) (2018)
- [Backpropagation Algorithms and Reservoir Computing in Recurrent Neural Networks for the Forecasting of Complex Spatiotemporal Dynamics](https://mlsanity.com/paper/1910.05266/) (2019)
- [Robust Learning with Jacobian Regularization](https://mlsanity.com/paper/1908.02729/) (2019)
- [Noisy Recurrent Neural Networks](https://mlsanity.com/paper/2102.04877/) (2021)
- [Combining Machine Learning with Knowledge-Based Modeling for Scalable Forecasting and Subgrid-Scale Closure of Large, Complex, Spatiotemporal Systems](https://mlsanity.com/paper/2002.05514/) (2020)
- [Model-free inference of unseen attractors: Reconstructing phase space features from a single noisy trajectory using reservoir computing](https://mlsanity.com/paper/2108.04074/) (2021)
- [Learning unseen coexisting attractors](https://mlsanity.com/paper/2207.14133/) (2022)
- [Deep Adversarial Koopman Model for Reaction-Diffusion systems](https://mlsanity.com/paper/2006.05547/) (2020)

## Built on by (most cited first, 5 of 5)

- [A non-intrusive machine learning framework for debiasing long-time coarse resolution climate simulations and quantifying rare events statistics](https://mlsanity.com/paper/2402.18484/) (2024)
- [A probabilistic framework for learning non-intrusive corrections to long-time climate simulations from short-time training data](https://mlsanity.com/paper/2408.02688/) (2024)
- [GEN2: A Generative Prediction-Correction Framework for Long-time Emulations of Spatially-Resolved Climate Extremes](https://mlsanity.com/paper/2508.15196/) (2025)
- [PT: A Plain Transformer is Good Hospital Readmission Predictor](https://mlsanity.com/paper/2412.12909/) (2024)
- [Eigenanalysis framework for autoregressive neural emulators of multi-scale chaotic dynamics](https://mlsanity.com/paper/2608.16084/) (2026)

## Abstract (arXiv)

Recent work has shown that machine learning (ML) models can be trained to accurately forecast the dynamics of unknown chaotic dynamical systems. Short-term predictions of the state evolution and long-term predictions of the statistical patterns of the dynamics (``climate'') can be produced by employing a feedback loop, whereby the model is trained to predict forward one time step, then the model output is used as input for multiple time steps. In the absence of mitigating techniques, however, this technique can result in artificially rapid error growth. In this article, we systematically examine the technique of adding noise to the ML model input during training to promote stability and improve prediction accuracy. Furthermore, we introduce Linearized Multi-Noise Training (LMNT), a regularization technique that deterministically approximates the effect of many small, independent noise realizations added to the model input during training. Our case study uses reservoir computing, a machine-learning method using recurrent neural networks, to predict the spatiotemporal chaotic Kuramoto-Sivashinsky equation. We find that reservoir computers trained with noise or with LMNT produce climate predictions that appear to be indefinitely stable and have a climate very similar to the true system, while reservoir computers trained without regularization are unstable. Compared with other regularization techniques that yield stability in some cases, we find that both short-term and climate predictions from reservoir computers trained with noise or with LMNT are substantially more accurate. Finally, we show that the deterministic aspect of our LMNT regularization facilitates fast hyperparameter tuning when compared to training with noise.