mlsanityData to 7 Oct 2026

A Random Matrix Perspective of Echo State Networks: From Precise Bias--Variance Characterization to Optimal Regularization

arXiv 2509.22011 ↗First posted 26 Sep 2025stat.ML cs.LG math.STcited by 0

Not on the map. Why?

Abstract (arXiv)

We present a rigorous asymptotic analysis of Echo State Networks (ESNs) in a teacher student setting with a linear teacher with oracle weights. Leveraging random matrix theory, we derive closed form expressions for the asymptotic bias, variance, and mean-squared error (MSE) as functions of the input statistics, the oracle vector, and the ridge regularization parameter. The analysis reveals two key departures from classical ridge regression: (i) ESNs do not exhibit double descent, and (ii) ESNs attain lower MSE when both the number of training samples and the teacher memory length are limited. We further provide an explicit formula for the optimal regularization in the identity input covariance case, and propose an efficient numerical scheme to compute the optimum in the general case. Together, these results offer interpretable theory and practical guidelines for tuning ESNs, helping reconcile recent empirical observations with provable performance guarantees

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 10 of its 22 references that are in the map. First posted on arXiv 26 Sep 2025.

Builds on

10 of 22 references.

References · 10

  1. 2018Echo state networks are universalcited by 86
  2. 2020A Random Matrix Analysis of Random Fourier Features: Beyond the Gaussian Kernel, a Precise Phase Transition, and the Corresponding Double Descentcited by 69
  3. 2020Random Matrix Theory Proves that Deep Learning Representations of GAN-data Behave as Gaussian Mixturescited by 47
  4. 2023High-dimensional analysis of double descent for linear regression with random projectionscited by 47
  5. 2024Scaling and renormalization in high-dimensional regressioncited by 45
  6. 2019Efficient Optimization of Echo State Networks for Time Series Datasetscited by 10
  7. 2018On the Statistical Challenges of Echo State Networks and Some Potential Remediescited by 7
  8. 2016Training Echo State Networks with Regularization through Dimensionality Reductioncited by 4
  9. 2022Hyperparameter Tuning in Echo State Networkscited by 2
  10. 2024Reservoir computing for system identification and predictive control with limited datacited by 2

Built on by

Not cited yet.