# Approximation Bounds for Random Neural Networks and Reservoir Systems

arXiv: https://arxiv.org/abs/2002.05933 · HTML: https://mlsanity.com/paper/2002.05933/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Feb 2020.

## Built on by (most cited first, 30 of 52)

- [Random feature neural networks learn Black-Scholes type PDEs without curse of dimensionality](https://mlsanity.com/paper/2106.08900/) (2021)
- [Universal Approximation Under Constraints is Possible with Transformers](https://mlsanity.com/paper/2110.03303/) (2021)
- [Error Bounds for Learning with Vector-Valued Random Features](https://mlsanity.com/paper/2305.17170/) (2023)
- [Operator Learning Using Random Features: A Tool for Scientific Computing](https://mlsanity.com/paper/2408.06526/) (2024)
- [A Brief Survey on the Approximation Theory for Sequence Modelling](https://mlsanity.com/paper/2302.13752/) (2023)
- [Universal approximation property of Banach space-valued random feature models including random neural networks](https://mlsanity.com/paper/2312.08410/) (2023)
- [Full error analysis of the random deep splitting method for nonlinear parabolic PDEs and PIDEs](https://mlsanity.com/paper/2405.05192/) (2024)
- [Non-asymptotic estimates for TUSLA algorithm for non-convex learning with applications to neural networks with ReLU activation function](https://mlsanity.com/paper/2107.08649/) (2021)
- [Chaotic Hedging with Iterated Integrals and Neural Networks](https://mlsanity.com/paper/2209.10166/) (2022)
- [Reservoir kernels and Volterra series](https://mlsanity.com/paper/2212.14641/) (2022)
- [A Theoretical Analysis of the Test Error of Finite-Rank Kernel Ridge Regression](https://mlsanity.com/paper/2310.00987/) (2023)
- [Convergence of Gradient Descent for Recurrent Neural Networks: A Nonasymptotic Analysis](https://mlsanity.com/paper/2402.12241/) (2024)
- [Infinite-dimensional reservoir computing](https://mlsanity.com/paper/2304.00490/) (2023)
- [Solving stochastic partial differential equations using neural networks in the Wiener chaos expansion](https://mlsanity.com/paper/2411.03384/) (2024)
- [Error Bounds of the Invariant Statistics in Machine Learning of Ergodic Ito Diffusions](https://mlsanity.com/paper/2105.10102/) (2021)
- [Universality and approximation bounds for echo state networks with random weights](https://mlsanity.com/paper/2206.05669/) (2022)
- [Langevin dynamics based algorithm e-THvarepsilonO POULA for stochastic optimization problems with discontinuous stochastic gradient](https://mlsanity.com/paper/2210.13193/) (2022)
- [How more data can hurt: Instability and regularization in next-generation reservoir computing](https://mlsanity.com/paper/2407.08641/) (2024)
- [Feedback-driven recurrent quantum neural network universality](https://mlsanity.com/paper/2506.16332/) (2025)
- [Transformers Can Solve Non-Linear and Non-Markovian Filtering Problems in Continuous Time For Conditionally Gaussian Signals](https://mlsanity.com/paper/2310.19603/) (2023)
- [Universal randomised signatures for generative time series modelling](https://mlsanity.com/paper/2406.10214/) (2024)
- [Random Feature Representation Boosting](https://mlsanity.com/paper/2501.18283/) (2025)
- [RandNet-Parareal: a time-parallel PDE solver using Random Neural Networks](https://mlsanity.com/paper/2411.06225/) (2024)
- [Stochastic dynamics learning with state-space systems](https://mlsanity.com/paper/2508.07876/) (2025)
- [Regret-Optimal Federated Transfer Learning for Kernel Regression with Applications in American Option Pricing](https://mlsanity.com/paper/2309.04557/) (2023)
- [Approximation Rates and VC-Dimension Bounds for (P)ReLU MLP Mixture of Experts](https://mlsanity.com/paper/2402.03460/) (2024)
- [Expressivity of Neural Networks with Random Weights and Learned Biases](https://mlsanity.com/paper/2407.00957/) (2024)
- [Fading memory and the convolution theorem](https://mlsanity.com/paper/2408.07386/) (2024)
- [A new local time-decoupled squared Wasserstein-2 method for training stochastic neural networks to reconstruct uncertain parameters in dynamical systems](https://mlsanity.com/paper/2503.05068/) (2025)
- [Path classification by stochastic linear recurrent neural networks](https://mlsanity.com/paper/2108.03090/) (2021)

## Abstract (arXiv)

This work studies approximation based on single-hidden-layer feedforward and recurrent neural networks with randomly generated internal weights. These methods, in which only the last layer of weights and a few hyperparameters are optimized, have been successfully applied in a wide range of static and dynamic learning problems. Despite the popularity of this approach in empirical tasks, important theoretical questions regarding the relation between the unknown function, the weight distribution, and the approximation rate have remained open. In this work it is proved that, as long as the unknown function, functional, or dynamical system is sufficiently regular, it is possible to draw the internal weights of the random (recurrent) neural network from a generic distribution (not depending on the unknown object) and quantify the error in terms of the number of neurons and the hyperparameters. In particular, this proves that echo state networks with randomly generated weights are capable of approximating a wide class of dynamical systems arbitrarily well and thus provides the first mathematical explanation for their empirically observed success at learning dynamical systems.