mlsanityData to 7 Oct 2026

How more data can hurt: Instability and regularization in next-generation reservoir computing

arXiv 2407.08641 ↗First posted 11 Jul 2024cs.LG cs.NE math.DScited by 5

Not on the map. Why?

Abstract (arXiv)

It has been found recently that more data can, counter-intuitively, hurt the performance of deep neural networks. Here, we show that a more extreme version of the phenomenon occurs in data-driven models of dynamical systems. To elucidate the underlying mechanism, we focus on next-generation reservoir computing (NGRC) -- a popular framework for learning dynamics from data. We find that, despite learning a better representation of the flow map with more training data, NGRC can adopt an ill-conditioned ``integrator'' and lose stability. We link this data-induced instability to the auxiliary dimensions created by the delayed states in NGRC. Based on these findings, we propose simple strategies to mitigate the instability, either by increasing regularization strength in tandem with data size, or by carefully introducing noise during training. Our results highlight the importance of proper regularization in data-driven modeling of dynamical systems.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 37 of its 76 references that are in the map. First posted on arXiv 11 Jul 2024.

Builds on

37 of 76 references.

References · 37

  1. 2018Neural Ordinary Differential Equationscited by 3,733
  2. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,806
  3. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  4. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  5. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  6. 2023Neural Operators for Accelerating Scientific Simulations and Designcited by 218
  7. 2019Two models of double descent for weak featurescited by 194
  8. 2021Next Generation Reservoir Computingcited by 160
  9. 2021Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolationcited by 130
  10. 2020Optimal Regularization Can Mitigate Double Descentcited by 107
  11. 2020Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regimecited by 95
  12. 2018Echo state networks are universalcited by 86
25 more
  1. 2018Hybrid Forecasting of Chaotic Processes: Using Machine Learning in Conjunction with a Knowledge-Based Modelcited by 57
  2. 2020Machine learning prediction of critical transition and system collapsecited by 54
  3. 2020Approximation Bounds for Random Neural Networks and Reservoir Systemscited by 52
  4. 2023Unifying Grokking and Double Descentcited by 49
  5. 2019More Data Can Hurt for Linear Regression: Sample-wise Double Descentcited by 37
  6. 2021A Framework for Machine Learning of Model Error in Dynamical Systemscited by 37
  7. 2023Double Descent Demystified: Identifying, Interpreting & Ablating the Sources of a Deep Learning Puzzlecited by 36
  8. 2024Out-of-Domain Generalization in Dynamical Systems Reconstructioncited by 33
  9. 2023Model scale versus domain knowledge in statistical forecasting of chaotic systemscited by 27
  10. 2021Anticipating synchronization with machine learningcited by 22
  11. 2023Generative learning for nonlinear dynamicscited by 22
  12. 2020Multifunctionality in a Reservoir Computercited by 21
  13. 2021Model-free inference of unseen attractors: Reconstructing phase space features from a single noisy trajectory using reservoir computingcited by 21
  14. 2019Differentiable reservoir computingcited by 20
  15. 2021Combining machine learning and data assimilation to forecast dynamical systems from noisy partial observationscited by 20
  16. 2022Learning Spatiotemporal Chaos Using Next-Generation Reservoir Computingcited by 20
  17. 2021Using Data Assimilation to Train a Hybrid Forecast System that Combines Machine-Learning and Knowledge-Based Componentscited by 18
  18. 2020Do Reservoir Computers Work Best at the Edge of Chaos?cited by 17
  19. 2022Learning unseen coexisting attractorscited by 12
  20. 2022Catch-22s of reservoir computingcited by 11
  21. 2024Controlling Chaos Using Edge Computing Hardwarecited by 11
  22. 2021Symmetry-Aware Reservoir Computingcited by 7
  23. 2023Controlling Chaotic Maps using Next-Generation Reservoir Computingcited by 7
  24. 2024Hybridizing Traditional and Next-Generation Reservoir Computing to Accurately and Efficiently Forecast Dynamical Systemscited by 6
  25. 2020Beyond Occam's Razor in System Identification: Double-Descent when Modeling Dynamicscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 2.

5 papers.

  1. 2024Zero-shot forecasting of chaotic systemscited by 25
  2. 2025On the emergence of numerical instabilities in Next Generation Reservoir Computingcited by 3
  3. 2025Next-Generation Reservoir Computing for Dynamical Inferencecited by 1
  4. 2026Theory and interpretability of Quantum Extreme Learning Machines: a Pauli-transfer matrix approachcited by 1
  5. 2026Flow map learning in nonlinear vector autoregressive models: influence of the feature-library structure on the training errorcited by 0