# How more data can hurt: Instability and regularization in next-generation reservoir computing

arXiv: https://arxiv.org/abs/2407.08641 · HTML: https://mlsanity.com/paper/2407.08641/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 76 references that are in the map.
- First posted on arXiv 11 Jul 2024.

## Builds on (most cited first, 30 of 37)

- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [Deep Double Descent: Where Bigger Models and More Data Hurt](https://mlsanity.com/paper/1912.02292/) (2019)
- [The generalization error of random features regression: Precise asymptotics and double descent curve](https://mlsanity.com/paper/1908.05355/) (2019)
- [Neural Operators for Accelerating Scientific Simulations and Design](https://mlsanity.com/paper/2309.15325/) (2023)
- [Two models of double descent for weak features](https://mlsanity.com/paper/1903.07571/) (2019)
- [Next Generation Reservoir Computing](https://mlsanity.com/paper/2106.07688/) (2021)
- [Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation](https://mlsanity.com/paper/2105.14368/) (2021)
- [Optimal Regularization Can Mitigate Double Descent](https://mlsanity.com/paper/2003.01897/) (2020)
- [Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regime](https://mlsanity.com/paper/2003.01054/) (2020)
- [Echo state networks are universal](https://mlsanity.com/paper/1806.00797/) (2018)
- [Hybrid Forecasting of Chaotic Processes: Using Machine Learning in Conjunction with a Knowledge-Based Model](https://mlsanity.com/paper/1803.04779/) (2018)
- [Machine learning prediction of critical transition and system collapse](https://mlsanity.com/paper/2012.01545/) (2020)
- [Approximation Bounds for Random Neural Networks and Reservoir Systems](https://mlsanity.com/paper/2002.05933/) (2020)
- [Unifying Grokking and Double Descent](https://mlsanity.com/paper/2303.06173/) (2023)
- [A Framework for Machine Learning of Model Error in Dynamical Systems](https://mlsanity.com/paper/2107.06658/) (2021)
- [More Data Can Hurt for Linear Regression: Sample-wise Double Descent](https://mlsanity.com/paper/1912.07242/) (2019)
- [Double Descent Demystified: Identifying, Interpreting & Ablating the Sources of a Deep Learning Puzzle](https://mlsanity.com/paper/2303.14151/) (2023)
- [Out-of-Domain Generalization in Dynamical Systems Reconstruction](https://mlsanity.com/paper/2402.18377/) (2024)
- [Model scale versus domain knowledge in statistical forecasting of chaotic systems](https://mlsanity.com/paper/2303.08011/) (2023)
- [Anticipating synchronization with machine learning](https://mlsanity.com/paper/2103.13358/) (2021)
- [Generative learning for nonlinear dynamics](https://mlsanity.com/paper/2311.04128/) (2023)
- [Multifunctionality in a Reservoir Computer](https://mlsanity.com/paper/2008.06348/) (2020)
- [Combining machine learning and data assimilation to forecast dynamical systems from noisy partial observations](https://mlsanity.com/paper/2108.03561/) (2021)
- [Model-free inference of unseen attractors: Reconstructing phase space features from a single noisy trajectory using reservoir computing](https://mlsanity.com/paper/2108.04074/) (2021)
- [Differentiable reservoir computing](https://mlsanity.com/paper/1902.06094/) (2019)
- [Learning Spatiotemporal Chaos Using Next-Generation Reservoir Computing](https://mlsanity.com/paper/2203.13294/) (2022)
- [Using Data Assimilation to Train a Hybrid Forecast System that Combines Machine-Learning and Knowledge-Based Components](https://mlsanity.com/paper/2102.07819/) (2021)
- [Do Reservoir Computers Work Best at the Edge of Chaos?](https://mlsanity.com/paper/2012.01409/) (2020)

## Built on by (most cited first, 5 of 5)

- [Zero-shot forecasting of chaotic systems](https://mlsanity.com/paper/2409.15771/) (2024)
- [On the emergence of numerical instabilities in Next Generation Reservoir Computing](https://mlsanity.com/paper/2505.00846/) (2025)
- [Next-Generation Reservoir Computing for Dynamical Inference](https://mlsanity.com/paper/2509.11338/) (2025)
- [Theory and interpretability of Quantum Extreme Learning Machines: a Pauli-transfer matrix approach](https://mlsanity.com/paper/2602.18377/) (2026)
- [Flow map learning in nonlinear vector autoregressive models: influence of the feature-library structure on the training error](https://mlsanity.com/paper/2605.31438/) (2026)

## Abstract (arXiv)

It has been found recently that more data can, counter-intuitively, hurt the performance of deep neural networks. Here, we show that a more extreme version of the phenomenon occurs in data-driven models of dynamical systems. To elucidate the underlying mechanism, we focus on next-generation reservoir computing (NGRC) -- a popular framework for learning dynamics from data. We find that, despite learning a better representation of the flow map with more training data, NGRC can adopt an ill-conditioned ``integrator'' and lose stability. We link this data-induced instability to the auxiliary dimensions created by the delayed states in NGRC. Based on these findings, we propose simple strategies to mitigate the instability, either by increasing regularization strength in tandem with data size, or by carefully introducing noise during training. Our results highlight the importance of proper regularization in data-driven modeling of dynamical systems.