# Using Echo State Networks to Approximate Value Functions for Control

arXiv: https://arxiv.org/abs/2102.06258 · HTML: https://mlsanity.com/paper/2102.06258/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 35 references that are in the map.
- First posted on arXiv 11 Feb 2021.

## Builds on (most cited first, 11 of 11)

- [Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator](https://mlsanity.com/paper/1801.05039/) (2018)
- [On the Sample Complexity of the Linear Quadratic Regulator](https://mlsanity.com/paper/1710.01688/) (2017)
- [Echo state networks are universal](https://mlsanity.com/paper/1806.00797/) (2018)
- [Approximation Bounds for Random Neural Networks and Reservoir Systems](https://mlsanity.com/paper/2002.05933/) (2020)
- [The Gap Between Model-Based and Model-Free Methods on the Linear Quadratic Regulator: An Asymptotic Viewpoint](https://mlsanity.com/paper/1812.03565/) (2018)
- [Echo State Networks trained by Tikhonov least squares are L2(μ) approximators of ergodic dynamical systems](https://mlsanity.com/paper/2005.06967/) (2020)
- [Learning the Linear Quadratic Regulator from Nonlinear Observations](https://mlsanity.com/paper/2010.03799/) (2020)
- [Differentiable reservoir computing](https://mlsanity.com/paper/1902.06094/) (2019)
- [General Value Function Networks](https://mlsanity.com/paper/1807.06763/) (2018)
- [On the Universal Transformation of Data-Driven Models to Control Systems](https://mlsanity.com/paper/2102.04722/) (2021)
- [Clustering piecewise stationary processes](https://mlsanity.com/paper/1906.10921/) (2019)

## Abstract (arXiv)

An Echo State Network (ESN) is a type of single-layer recurrent neural network with randomly-chosen internal weights and a trainable output layer. We prove under mild conditions that a sufficiently large Echo State Network can approximate the value function of a broad class of stochastic and deterministic control problems. Such control problems are generally non-Markovian. We describe how the ESN can form the basis for novel and computationally efficient reinforcement learning algorithms in a non-Markovian framework. We demonstrate this theory with two examples. In the first, we use an ESN to solve a deterministic, partially observed, control problem which is a simple game we call `Bee World'. In the second example, we consider a stochastic control problem inspired by a market making problem in mathematical finance. In both cases we can compare the dynamics of the algorithms with analytic solutions to show that even after only a single reinforcement policy iteration the algorithms arrive at a good policy.