# Complexity-calibrated Benchmarks for Machine Learning Reveal When Next-Generation Reservoir Computer Predictions Succeed and Mislead

arXiv: https://arxiv.org/abs/2303.14553 · HTML: https://mlsanity.com/paper/2303.14553/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 36 references that are in the map.
- First posted on arXiv 25 Mar 2023.

## Builds on (most cited first, 6 of 6)

- [A Critical Review of Recurrent Neural Networks for Sequence Learning](https://mlsanity.com/paper/1506.00019/) (2015)
- [Next Generation Reservoir Computing](https://mlsanity.com/paper/2106.07688/) (2021)
- [Theory of gating in recurrent neural networks](https://mlsanity.com/paper/2007.14823/) (2020)
- [Nearly Maximally Predictive Features and Their Dimensions](https://mlsanity.com/paper/1702.08565/) (2017)
- [Shannon Entropy Rate of Hidden Markov Processes](https://mlsanity.com/paper/2008.12886/) (2020)
- [Time Resolution Dependence of Information Measures for Spiking Neurons: Atoms, Scaling, and Universality](https://mlsanity.com/paper/1504.04756/) (2015)

## Built on by (most cited first, 2 of 2)

- [Generative learning for nonlinear dynamics](https://mlsanity.com/paper/2311.04128/) (2023)
- [Ultimate limit on learning non-Markovian behavior: Fisher information rate and excess information](https://mlsanity.com/paper/2310.03968/) (2023)

## Abstract (arXiv)

Recurrent neural networks are used to forecast time series in finance, climate, language, and from many other domains. Reservoir computers are a particularly easily trainable form of recurrent neural network. Recently, a "next-generation" reservoir computer was introduced in which the memory trace involves only a finite number of previous symbols. We explore the inherent limitations of finite-past memory traces in this intriguing proposal. A lower bound from Fano's inequality shows that, on highly non-Markovian processes generated by large probabilistic state machines, next-generation reservoir computers with reasonably long memory traces have an error probability that is at least ~ 60% higher than the minimal attainable error probability in predicting the next observation. More generally, it appears that popular recurrent neural networks fall far short of optimally predicting such complex processes. These results highlight the need for a new generation of optimized recurrent neural network architectures. Alongside this finding, we present concentration-of-measure results for randomly-generated but complex processes. One conclusion is that large probabilistic state machines -- specifically, large ε-machines -- are key to generating challenging and structurally-unbiased stimuli for ground-truthing recurrent neural network architectures.