# Metric Entropy Limits on Recurrent Neural Network Learning of Linear Dynamical Systems

arXiv: https://arxiv.org/abs/2105.02556 · HTML: https://mlsanity.com/paper/2105.02556/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 46 references that are in the map.
- First posted on arXiv 6 May 2021.

## Builds on (most cited first, 11 of 11)

- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Deep Neural Network Approximation Theory](https://mlsanity.com/paper/1901.02220/) (2019)
- [Optimal Approximation with Sparsely Connected Deep Neural Networks](https://mlsanity.com/paper/1705.01714/) (2017)
- [Gradient Descent Learns Linear Dynamical Systems](https://mlsanity.com/paper/1609.05191/) (2016)
- [A proof that artificial neural networks overcome the curse of dimensionality in the numerical approximation of Black-Scholes partial differential equations](https://mlsanity.com/paper/1809.02362/) (2018)
- [Analysis of the Generalization Error: Empirical Risk Minimization over Deep Artificial Neural Networks Overcomes the Curse of Dimensionality in the Numerical Approximation of Black-Scholes Partial Differential Equations](https://mlsanity.com/paper/1809.03062/) (2018)
- [On the Curse of Memory in Recurrent Neural Networks: Approximation and Optimization Analysis](https://mlsanity.com/paper/2009.07799/) (2020)
- [Affine symmetries and neural network identifiability](https://mlsanity.com/paper/2006.11727/) (2020)
- [Neural network identifiability for a family of sigmoidal nonlinearities](https://mlsanity.com/paper/1906.06994/) (2019)

## Built on by (most cited first, 4 of 4)

- [Partial Differential Equations Meet Deep Neural Networks: A Survey](https://mlsanity.com/paper/2211.05567/) (2022)
- [Designing Universal Causal Deep Learning Models: The Geometric (Hyper)Transformer](https://mlsanity.com/paper/2201.13094/) (2022)
- [Designing Universal Causal Deep Learning Models: The Case of Infinite-Dimensional Dynamical Systems from Stochastic Analysis](https://mlsanity.com/paper/2210.13300/) (2022)
- [Approximation Rates and VC-Dimension Bounds for (P)ReLU MLP Mixture of Experts](https://mlsanity.com/paper/2402.03460/) (2024)

## Abstract (arXiv)

One of the most influential results in neural network theory is the universal approximation theorem [1, 2, 3] which states that continuous functions can be approximated to within arbitrary accuracy by single-hidden-layer feedforward neural networks. The purpose of this paper is to establish a result in this spirit for the approximation of general discrete-time linear dynamical systems - including time-varying systems - by recurrent neural networks (RNNs). For the subclass of linear time-invariant (LTI) systems, we devise a quantitative version of this statement. Specifically, measuring the complexity of the considered class of LTI systems through metric entropy according to [4], we show that RNNs can optimally learn - or identify in system-theory parlance - stable LTI systems. For LTI systems whose input-output relation is characterized through a difference equation, this means that RNNs can learn the difference equation from input-output traces in a metric-entropy optimal manner.