# Universality and approximation bounds for echo state networks with random weights

arXiv: https://arxiv.org/abs/2206.05669 · HTML: https://mlsanity.com/paper/2206.05669/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 40 references that are in the map.
- First posted on arXiv 12 Jun 2022.

## Builds on (most cited first, 11 of 11)

- [Echo state networks are universal](https://mlsanity.com/paper/1806.00797/) (2018)
- [Approximation by Combinations of ReLU and Squared ReLU Ridge Functions with  ℓ1  and  ℓ0  Controls](https://mlsanity.com/paper/1607.07819/) (2016)
- [A Priori Estimates of the Population Risk for Two-layer Neural Networks](https://mlsanity.com/paper/1810.06397/) (2018)
- [Representation formulas and pointwise properties for Barron functions](https://mlsanity.com/paper/2006.05982/) (2020)
- [Approximation Bounds for Random Neural Networks and Reservoir Systems](https://mlsanity.com/paper/2002.05933/) (2020)
- [Reservoir Computing Universality With Stochastic Inputs](https://mlsanity.com/paper/1807.02621/) (2018)
- [Universal discrete-time reservoir computers with stochastic inputs and linear readouts using non-homogeneous state-affine systems](https://mlsanity.com/paper/1712.00754/) (2017)
- [Fading memory echo state networks are universal](https://mlsanity.com/paper/2010.12047/) (2020)
- [Risk bounds for reservoir computing](https://mlsanity.com/paper/1910.13886/) (2019)
- [Differentiable reservoir computing](https://mlsanity.com/paper/1902.06094/) (2019)
- [Universal Approximation of Input-Output Maps by Temporal Convolutional Nets](https://mlsanity.com/paper/1906.09211/) (2019)

## Built on by (most cited first, 5 of 5)

- [On the Optimal Expressive Power of ReLU DNNs and Its Application in Approximation with Kolmogorov Superposition Theorem](https://mlsanity.com/paper/2308.05509/) (2023)
- [ResCP: Reservoir Conformal Prediction for Time Series Forecasting](https://mlsanity.com/paper/2510.05060/) (2025)
- [Stochastic Reservoir Computers](https://mlsanity.com/paper/2405.12382/) (2024)
- [Universality of reservoir systems with recurrent neural networks](https://mlsanity.com/paper/2403.01900/) (2024)
- [Online Federation For Mixtures of Proprietary Agents with Black-Box Encoders](https://mlsanity.com/paper/2505.00216/) (2025)

## Abstract (arXiv)

We study the uniform approximation of echo state networks with randomly generated internal weights. These models, in which only the readout weights are optimized during training, have made empirical success in learning dynamical systems. Recent results showed that echo state networks with ReLU activation are universal. In this paper, we give an alternative construction and prove that the universality holds for general activation functions. Specifically, our main result shows that, under certain condition on the activation function, there exists a sampling procedure for the internal weights so that the echo state network can approximate any continuous casual time-invariant operators with high probability. In particular, for ReLU activation, we give explicit construction for these sampling procedures. We also quantify the approximation error of the constructed ReLU echo state networks for sufficiently regular operators.