# Echo State Networks trained by Tikhonov least squares are L2(μ) approximators of ergodic dynamical systems

arXiv: https://arxiv.org/abs/2005.06967 · HTML: https://mlsanity.com/paper/2005.06967/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 May 2020.

## Built on by (most cited first, 21 of 21)

- [Next Generation Reservoir Computing](https://mlsanity.com/paper/2106.07688/) (2021)
- [A Survey on Reservoir Computing and its Interdisciplinary Applications Beyond Traditional Machine Learning](https://mlsanity.com/paper/2307.15092/) (2023)
- [Learning strange attractors with reservoir systems](https://mlsanity.com/paper/2108.05024/) (2021)
- [Catch-22s of reservoir computing](https://mlsanity.com/paper/2210.10211/) (2022)
- [Embedding Theory of Reservoir Computing and Reducing Reservoir Network Using Time Delays](https://mlsanity.com/paper/2303.09042/) (2023)
- [Reservoir kernels and Volterra series](https://mlsanity.com/paper/2212.14641/) (2022)
- [Data-driven prediction and control of extreme events in a chaotic flow](https://mlsanity.com/paper/2204.11682/) (2022)
- [Quantum Next Generation Reservoir Computing: An Efficient Quantum Algorithm for Forecasting Quantum Dynamics](https://mlsanity.com/paper/2308.14239/) (2023)
- [Gradient-free optimization of chaotic acoustics with reservoir computing](https://mlsanity.com/paper/2106.09780/) (2021)
- [Modelling spatiotemporal turbulent dynamics with the convolutional autoencoder echo state network](https://mlsanity.com/paper/2211.11379/) (2022)
- [On the emergence of numerical instabilities in Next Generation Reservoir Computing](https://mlsanity.com/paper/2505.00846/) (2025)
- [Using Connectome Features to Constrain Echo State Networks](https://mlsanity.com/paper/2206.02094/) (2022)
- [Expressivity of Neural Networks with Random Weights and Learned Biases](https://mlsanity.com/paper/2407.00957/) (2024)
- [Minimal Deterministic Echo State Networks Outperform Random Reservoirs in Learning Chaotic Dynamics](https://mlsanity.com/paper/2507.06050/) (2025)
- [Unsupervised Learning in Echo State Networks for Input Reconstruction](https://mlsanity.com/paper/2501.11409/) (2025)
- [Using Echo State Networks to Approximate Value Functions for Control](https://mlsanity.com/paper/2102.06258/) (2021)
- [Variability of echo state network prediction horizon for partially observed dynamical systems](https://mlsanity.com/paper/2306.10797/) (2023)
- [Universal Approximation of Linear Time-Invariant (LTI) Systems through RNNs: Power of Randomness in Reservoir Computing](https://mlsanity.com/paper/2308.02464/) (2023)
- [Estimation of AMOC transition probabilities using a machine learning based rare-event algorithm](https://mlsanity.com/paper/2401.10800/) (2024)
- [Phase space integrity in neural network models of Hamiltonian dynamics: A Lagrangian descriptor approach](https://mlsanity.com/paper/2604.00473/) (2026)
- [Data-Specific Hyper-Parameter Design: A Paradigm Shift in Reservoir Computing](https://mlsanity.com/paper/2605.25221/) (2026)

## Abstract (arXiv)

Echo State Networks (ESNs) are a class of single-layer recurrent neural networks with randomly generated internal weights, and a single layer of tuneable outer weights, which are usually trained by regularised linear least squares regression. Remarkably, ESNs still enjoy the universal approximation property despite the training procedure being entirely linear. In this paper, we prove that an ESN trained on a sequence of observations from an ergodic dynamical system (with invariant measure μ) using Tikhonov least squares regression against a set of targets, will approximate the target function in the L2(μ) norm. In the special case that the targets are future observations, the ESN is learning the next step map, which allows time series forecasting. We demonstrate the theory numerically by training an ESN using Tikhonov least squares on a sequence of scalar observations of the Lorenz system.