# Robust Optimization and Validation of Echo State Networks for learning chaotic dynamics

arXiv: https://arxiv.org/abs/2103.03174 · HTML: https://mlsanity.com/paper/2103.03174/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 67 references that are in the map.
- First posted on arXiv 9 Feb 2021.

## Builds on (most cited first, 21 of 21)

- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Machine Learning for Fluid Mechanics](https://mlsanity.com/paper/1905.11075/) (2019)
- [Data-Driven Forecasting of High-Dimensional Chaotic Systems with Long Short-Term Memory Networks](https://mlsanity.com/paper/1802.07486/) (2018)
- [Backpropagation Algorithms and Reservoir Computing in Recurrent Neural Networks for the Forecasting of Complex Spatiotemporal Dynamics](https://mlsanity.com/paper/1910.05266/) (2019)
- [Echo state networks are universal](https://mlsanity.com/paper/1806.00797/) (2018)
- [Hybrid Forecasting of Chaotic Processes: Using Machine Learning in Conjunction with a Knowledge-Based Model](https://mlsanity.com/paper/1803.04779/) (2018)
- [Forecasting Chaotic Systems with Very Low Connectivity Reservoir Computers](https://mlsanity.com/paper/1910.00659/) (2019)
- [Model-free prediction of spatiotemporal dynamical systems with recurrent neural networks: Role of network spectral radius](https://mlsanity.com/paper/1910.04426/) (2019)
- [Fading memory echo state networks are universal](https://mlsanity.com/paper/2010.12047/) (2020)
- [Combining Machine Learning with Knowledge-Based Modeling for Scalable Forecasting and Subgrid-Scale Closure of Large, Complex, Spatiotemporal Systems](https://mlsanity.com/paper/2002.05514/) (2020)
- [Machine-learning inference of fluid variables from data using reservoir computing](https://mlsanity.com/paper/1805.09917/) (2018)
- [Physics-Informed Echo State Networks](https://mlsanity.com/paper/2011.02280/) (2020)
- [Bayesian optimization of hyper-parameters in reservoir computing](https://mlsanity.com/paper/1611.05193/) (2016)
- [Data-driven prediction of a multi-scale Lorenz 96 chaotic system using deep learning methods: Reservoir computing, ANN, and RNN-LSTM](https://mlsanity.com/paper/1906.08829/) (2019)
- [Automatic-differentiated Physics-Informed Echo State Network (API-ESN)](https://mlsanity.com/paper/2101.00002/) (2020)
- [Physics-Informed Echo State Networks for Chaotic Systems Forecasting](https://mlsanity.com/paper/1906.11122/) (2019)
- [Efficient Cross-Validation of Echo State Networks](https://mlsanity.com/paper/1908.08450/) (2019)
- [Learning Hidden States in a Chaotic System: A Physics-Informed Echo State Network Approach](https://mlsanity.com/paper/2001.02982/) (2020)
- [Learning ergodic averages in chaotic systems](https://mlsanity.com/paper/2001.04027/) (2020)
- [A physics-aware machine to predict extreme events in turbulence](https://mlsanity.com/paper/1912.10994/) (2019)

## Built on by (most cited first, 27 of 27)

- [When are dynamical systems learned from time series data statistically accurate?](https://mlsanity.com/paper/2411.06311/) (2024)
- [Optimal training of finitely-sampled quantum reservoir computers for forecasting of chaotic dynamics](https://mlsanity.com/paper/2409.01394/) (2024)
- [Reconstruction, forecasting, and stability of chaotic dynamics from partial data](https://mlsanity.com/paper/2305.15111/) (2023)
- [Chaotic attractor reconstruction using small reservoirs -- the influence of topology](https://mlsanity.com/paper/2402.16888/) (2024)
- [Hyperparameter Optimization for Randomized Algorithms: A Case Study on Random Features](https://mlsanity.com/paper/2407.00584/) (2024)
- [Stability analysis of chaotic systems in latent spaces](https://mlsanity.com/paper/2410.00480/) (2024)
- [Data-driven prediction and control of extreme events in a chaotic flow](https://mlsanity.com/paper/2204.11682/) (2022)
- [Gradient-free optimization of chaotic acoustics with reservoir computing](https://mlsanity.com/paper/2106.09780/) (2021)
- [Modelling spatiotemporal turbulent dynamics with the convolutional autoencoder echo state network](https://mlsanity.com/paper/2211.11379/) (2022)
- [Towards Trustworthy Machine Learning in Production: An Overview of the Robustness in MLOps Approach](https://mlsanity.com/paper/2410.21346/) (2024)
- [Data-Assimilated Model-Based Reinforcement Learning for Partially Observed Chaotic Flows](https://mlsanity.com/paper/2504.16588/) (2025)
- [Statistical prediction of extreme events from small datasets](https://mlsanity.com/paper/2201.08294/) (2022)
- [Using Connectome Features to Constrain Echo State Networks](https://mlsanity.com/paper/2206.02094/) (2022)
- [Physics-aware deep learning framework for linear elasticity](https://mlsanity.com/paper/2302.09668/) (2023)
- [Control-aware echo state networks (Ca-ESN) for the suppression of extreme events](https://mlsanity.com/paper/2308.03095/) (2023)
- [Online model learning with data-assimilated reservoir computers](https://mlsanity.com/paper/2504.16767/) (2025)
- [Minimal Deterministic Echo State Networks Outperform Random Reservoirs in Learning Chaotic Dynamics](https://mlsanity.com/paper/2507.06050/) (2025)
- [Data-assimilated model-informed reinforcement learning](https://mlsanity.com/paper/2506.01755/) (2025)
- [Survey on Deep Fuzzy Systems in regression applications: a view on interpretability](https://mlsanity.com/paper/2209.04230/) (2022)
- [Variability of echo state network prediction horizon for partially observed dynamical systems](https://mlsanity.com/paper/2306.10797/) (2023)
- [Adjoint Sensitivities of Chaotic Flows without Adjoint Solvers: A Data-Driven Approach](https://mlsanity.com/paper/2404.12315/) (2024)
- [Inferring stability properties of chaotic systems on autoencoders' latent spaces](https://mlsanity.com/paper/2410.18003/) (2024)
- [Using Echo-State Networks to Reproduce Rare Events in Chaotic Systems](https://mlsanity.com/paper/2505.16208/) (2025)
- [Attractor learning for spatiotemporally chaotic dynamical systems using echo state networks with transfer learning](https://mlsanity.com/paper/2505.24099/) (2025)
- [Fractal and Chaotic Activation Functions in Echo State Networks: Preprocessing Topology Governs the Echo State Property](https://mlsanity.com/paper/2512.14675/) (2025)
- [Distinguishing indistinguishable attractors: Unsupervised anomaly detection with reservoir computers](https://mlsanity.com/paper/2606.21322/) (2026)
- [Free-Probability Kernels for Zero-Rollout Hyperparameter Selection in Reservoir Computing](https://mlsanity.com/paper/2608.20998/) (2026)

## Abstract (arXiv)

An approach to the time-accurate prediction of chaotic solutions is by learning temporal patterns from data. Echo State Networks (ESNs), which are a class of Reservoir Computing, can accurately predict the chaotic dynamics well beyond the predictability time. Existing studies, however, also showed that small changes in the hyperparameters may markedly affect the network's performance. The aim of this paper is to assess and improve the robustness of Echo State Networks for the time-accurate prediction of chaotic solutions. The goal is three-fold. First, we investigate the robustness of routinely used validation strategies. Second, we propose the Recycle Validation, and the chaotic versions of existing validation strategies, to specifically tackle the forecasting of chaotic systems. Third, we compare Bayesian optimization with the traditional Grid Search for optimal hyperparameter selection. Numerical tests are performed on two prototypical nonlinear systems that have both chaotic and quasiperiodic solutions. Both model-free and model-informed Echo State Networks are analysed. By comparing the network's robustness in learning chaotic versus quasiperiodic solutions, we highlight fundamental challenges in learning chaotic solutions. The proposed validation strategies, which are based on the dynamical systems properties of chaotic time series, are shown to outperform the state-of-the-art validation strategies. Because the strategies are principled-they are based on chaos theory such as the Lyapunov time-they can be applied to other Recurrent Neural Networks architectures with little modification. This work opens up new possibilities for the robust design and application of Echo State Networks, and Recurrent Neural Networks, to the time-accurate prediction of chaotic systems.