mlsanityData to 7 Oct 2026

Learning and Generalization in RNNs

arXiv 2106.00047 ↗First posted 31 May 2021cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Simple recurrent neural networks (RNNs) and their more advanced cousins LSTMs etc. have been very successful in sequence modeling. Their theoretical understanding, however, is lacking and has not kept pace with the progress for feedforward networks, where a reasonably complete understanding in the special case of highly overparametrized one-hidden-layer networks has emerged. In this paper, we make progress towards remedying this situation by proving that RNNs can learn functions of sequences. In contrast to the previous work that could only deal with functions of sequences that are sums of functions of individual tokens in the sequence, we allow general functions. Conceptually and technically, we introduce new ideas which enable us to extract information from the hidden state of the RNN in our proofs -- addressing a crucial weakness in previous work. We illustrate our results on some regular language recognition problems.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 28 of its 50 references that are in the map. First posted on arXiv 31 May 2021.

Builds on

28 of 50 references.

References · 28

  1. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  4. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  5. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  6. 2020Time Series Forecasting With Deep Learning: A Surveycited by 457
  7. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 404
  8. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  9. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 329
  10. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
  11. 2019Linearized two-layers neural networks in high dimensioncited by 155
  12. 2019Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processescited by 148
16 more
  1. 2018On the Practical Computational Power of Finite Precision RNNs for Language Recognitioncited by 117
  2. 2018Echo state networks are universalcited by 86
  3. 2018On the Convergence Rate of Training Recurrent Neural Networkscited by 82
  4. 2020O(n) Connections are Expressive Enough: Universal Approximability of Sparse Transformerscited by 82
  5. 2016Gradient Descent Learns Linear Dynamical Systemscited by 78
  6. 2020The Recurrent Neural Tangent Kernelcited by 67
  7. 2018Stable Recurrent Modelscited by 65
  8. 2020A Formal Hierarchy of RNN Architecturescited by 61
  9. 2019Sequential Neural Networks as Automatacited by 54
  10. 2019On Generalization Bounds of a Family of Recurrent Neural Networkscited by 42
  11. 2019Reverse engineering recurrent networks for sentiment classification reveals line attractor dynamicscited by 38
  12. 2019On the Computational Power of RNNscited by 35
  13. 2019Can SGD Learn Recurrent Neural Networks with Provable Generalization?cited by 17
  14. 2018Stochastic Gradient Descent Learns State Equations with Nonlinear Activationscited by 16
  15. 2021Implicit Bias of Linear RNNscited by 11
  16. 2021Formal Language Theory Meets Modern NLPcited by 9

Built on by

Not cited yet.