mlsanityData to 7 Oct 2026

Convergence of Gradient Descent for Recurrent Neural Networks: A Nonasymptotic Analysis

arXiv 2402.12241 ↗First posted 19 Feb 2024cs.LG math.OC stat.MLcited by 7

Not on the map. Why?

Abstract (arXiv)

We analyze recurrent neural networks with diagonal hidden-to-hidden weight matrices, trained with gradient descent in the supervised learning setting, and prove that gradient descent can achieve optimality without massive overparameterization. Our in-depth nonasymptotic analysis (i) provides improved bounds on the network size m in terms of the sequence length T, sample size n and ambient dimension d, and (ii) identifies the significant impact of long-term dependencies in the dynamical system on the convergence and network width bounds characterized by a cutoff point that depends on the Lipschitz continuity of the activation function. Remarkably, this analysis reveals that an appropriately-initialized recurrent neural network trained with n samples can achieve optimality with a network size m that scales only logarithmically with n. This sharply contrasts with the prior works that require high-order polynomial dependency of m on n to establish strong regularity conditions. Our results are based on an explicit characterization of the class of dynamical systems that can be approximated and learned by recurrent neural networks via norm-constrained transportation mappings, and establishing local smoothness properties of the hidden state with respect to the learnable parameters.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 24 of its 43 references that are in the map. First posted on arXiv 19 Feb 2024.

Builds on

24 of 43 references.

References · 24

  1. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  4. 2018On Lazy Training in Differentiable Programmingcited by 671
  5. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 666
  6. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  7. 2013Generating Sequences With Recurrent Neural Networkscited by 549
  8. 2014Recurrent Models of Visual Attentioncited by 344
  9. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  10. 2013Maxout Networkscited by 186
  11. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
  12. 2019Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networkscited by 120
12 more
  1. 2018On the Convergence Rate of Training Recurrent Neural Networkscited by 82
  2. 2016Gradient Descent Learns Linear Dynamical Systemscited by 78
  3. 2020The Recurrent Neural Tangent Kernelcited by 67
  4. 2013How to Construct Deep Recurrent Neural Networkscited by 64
  5. 2020Approximation Bounds for Random Neural Networks and Reservoir Systemscited by 52
  6. 2019Neural tangent kernels, transportation mappings, and universal approximationcited by 30
  7. 2019Can SGD Learn Recurrent Neural Networks with Provable Generalization?cited by 17
  8. 2023Robust Implicit Regularization via Weight Normalizationcited by 10
  9. 2021Sample Complexity and Overparameterization Bounds for Temporal Difference Learning with Neural Network Approximationcited by 9
  10. 2022Recurrent Neural Networks and Universal Approximation of Bayesian Filterscited by 5
  11. 2023Kernel Limit for a Class of Recurrent Neural Networks Trained on Ergodic Data Sequencescited by 2
  12. 2021Path classification by stochastic linear recurrent neural networkscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 2.

7 papers.

  1. 2024Recurrent Natural Policy Gradient for POMDPscited by 8
  2. 2025A Theoretical Justification for Asymmetric Actor-Critic Algorithmscited by 5
  3. 2024A Riemannian Optimization Perspective of the Gauss-Newton Method for Feedforward Neural Networkscited by 2
  4. 2025Convergence of Stochastic Gradient Langevin Dynamics in the Lazy Training Regimecited by 0
  5. 2025Non-Asymptotic Optimization and Generalization Bounds for Stochastic Gauss-Newton in Overparameterized Modelscited by 0
  6. 2026Finite-Time Analysis of Gradient Descent for Shallow Transformerscited by 0
  7. 2026Learning Suffers More Than the Policy Class Under Partial Observability: A Closed-Form Analysiscited by 0