mlsanityData to 7 Oct 2026

Metric Entropy Limits on Recurrent Neural Network Learning of Linear Dynamical Systems

arXiv 2105.02556 ↗First posted 6 May 2021cs.LG cs.IT math.DScited by 4

Not on the map. Why?

Abstract (arXiv)

One of the most influential results in neural network theory is the universal approximation theorem [1, 2, 3] which states that continuous functions can be approximated to within arbitrary accuracy by single-hidden-layer feedforward neural networks. The purpose of this paper is to establish a result in this spirit for the approximation of general discrete-time linear dynamical systems - including time-varying systems - by recurrent neural networks (RNNs). For the subclass of linear time-invariant (LTI) systems, we devise a quantitative version of this statement. Specifically, measuring the complexity of the considered class of LTI systems through metric entropy according to [4], we show that RNNs can optimally learn - or identify in system-theory parlance - stable LTI systems. For LTI systems whose input-output relation is characterized through a difference equation, this means that RNNs can learn the difference equation from input-output traces in a metric-entropy optimal manner.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 11 of its 46 references that are in the map. First posted on arXiv 6 May 2021.

Builds on

11 of 46 references.

References · 11

  1. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  2. 2019DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operatorscited by 1,965
  3. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  4. 2019Deep Neural Network Approximation Theorycited by 92
  5. 2017Optimal Approximation with Sparsely Connected Deep Neural Networkscited by 90
  6. 2016Gradient Descent Learns Linear Dynamical Systemscited by 78
  7. 2018A proof that artificial neural networks overcome the curse of dimensionality in the numerical approximation of Black-Scholes partial differential equationscited by 55
  8. 2018Analysis of the Generalization Error: Empirical Risk Minimization over Deep Artificial Neural Networks Overcomes the Curse of Dimensionality in the Numerical Approximation of Black-Scholes Partial Differential Equationscited by 44
  9. 2020On the Curse of Memory in Recurrent Neural Networks: Approximation and Optimization Analysiscited by 26
  10. 2020Affine symmetries and neural network identifiabilitycited by 17
  11. 2019Neural network identifiability for a family of sigmoidal nonlinearitiescited by 14

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 0; 2024: 1.

4 papers.

  1. 2022Partial Differential Equations Meet Deep Neural Networks: A Surveycited by 25
  2. 2022Designing Universal Causal Deep Learning Models: The Geometric (Hyper)Transformercited by 22
  3. 2022Designing Universal Causal Deep Learning Models: The Case of Infinite-Dimensional Dynamical Systems from Stochastic Analysiscited by 22
  4. 2024Approximation Rates and VC-Dimension Bounds for (P)ReLU MLP Mixture of Expertscited by 2