# Exploiting Multiple Timescales in Hierarchical Echo State Networks

arXiv: https://arxiv.org/abs/2101.04223 · HTML: https://mlsanity.com/paper/2101.04223/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 43 references that are in the map.
- First posted on arXiv 11 Jan 2021.

## Builds on (most cited first, 7 of 7)

- [Towards Non-saturating Recurrent Units for Modelling Long-term Dependencies](https://mlsanity.com/paper/1902.06704/) (2019)
- [Determination of the edge of criticality in echo state networks through Fisher information maximization](https://mlsanity.com/paper/1603.03685/) (2016)
- [Optimal modularity and memory capacity of neural reservoirs](https://mlsanity.com/paper/1706.06511/) (2017)
- [A Comparative Study of Reservoir Computing for Temporal Signal Processing](https://mlsanity.com/paper/1401.2224/) (2014)
- [The difference between memory and prediction in linear recurrent networks](https://mlsanity.com/paper/1706.09382/) (2017)
- [Concentric ESN: Assessing the Effect of Modularity in Cycle Reservoirs](https://mlsanity.com/paper/1805.09244/) (2018)
- [Deep Echo State Networks for Diagnosis of Parkinson's Disease](https://mlsanity.com/paper/1802.06708/) (2018)

## Built on by (most cited first, 11 of 11)

- [Reservoir Computing with Diverse Timescales for Prediction of Multiscale Dynamics](https://mlsanity.com/paper/2108.09446/) (2021)
- [A perspective on physical reservoir computing with nanomagnetic devices](https://mlsanity.com/paper/2212.04851/) (2022)
- [Reservoir Computing Benchmarks: a tutorial review and critique](https://mlsanity.com/paper/2405.06561/) (2024)
- [EchoVPR: Echo State Networks for Visual Place Recognition](https://mlsanity.com/paper/2110.05572/) (2021)
- [A Dynamic Systems Approach to Modelling Human-Machine Rhythm Interaction](https://mlsanity.com/paper/2407.09538/) (2024)
- [A novel Reservoir Architecture for Periodic Time Series Prediction](https://mlsanity.com/paper/2405.10102/) (2024)
- [Mixed Delay/Nondelay Embeddings Based Neuromorphic Computing with Patterned Nanomagnet Arrays](https://mlsanity.com/paper/2412.04622/) (2024)
- [Heterogeneous Reservoir Computing Models for Persian Speech Recognition](https://mlsanity.com/paper/2205.12594/) (2022)
- [Prediction of Unobserved Bifurcation by Unsupervised Extraction of Slowly Time-Varying System Parameter Dynamics from Time Series Using Reservoir Computing](https://mlsanity.com/paper/2406.13995/) (2024)
- [Evolutionary Algorithm for Reservoir Learning and Yielding](https://mlsanity.com/paper/2605.30372/) (2026)
- [Explicit Interaction Architectures for Dynamical Learning: A Controlled Study of Structural Inductive Bias](https://mlsanity.com/paper/2606.19101/) (2026)

## Abstract (arXiv)

Echo state networks (ESNs) are a powerful form of reservoir computing that only require training of linear output weights whilst the internal reservoir is formed of fixed randomly connected neurons. With a correctly scaled connectivity matrix, the neurons' activity exhibits the echo-state property and responds to the input dynamics with certain timescales. Tuning the timescales of the network can be necessary for treating certain tasks, and some environments require multiple timescales for an efficient representation. Here we explore the timescales in hierarchical ESNs, where the reservoir is partitioned into two smaller linked reservoirs with distinct properties. Over three different tasks (NARMA10, a reconstruction task in a volatile environment, and psMNIST), we show that by selecting the hyper-parameters of each partition such that they focus on different timescales, we achieve a significant performance improvement over a single ESN. Through a linear analysis, and under the assumption that the timescales of the first partition are much shorter than the second's (typically corresponding to optimal operating conditions), we interpret the feedforward coupling of the partitions in terms of an effective representation of the input signal, provided by the first partition to the second, whereby the instantaneous input signal is expanded into a weighted combination of its time derivatives. Furthermore, we propose a data-driven approach to optimise the hyper-parameters through a gradient descent optimisation method that is an online approximation of backpropagation through time. We demonstrate the application of the online learning rule across all the tasks considered.