mlsanityData to 14 Oct 2026

On the Convergence Rate of Training Recurrent Neural Networks

arXiv 1810.12065 ↗First posted 29 Oct 2018cs.LG cs.DS cs.NEcited by 82

Not on the map. Why?

Abstract (arXiv)

How can local-search methods such as stochastic gradient descent (SGD) avoid bad local minima in training multi-layer neural networks? Why can they fit random labels even given non-convex and non-smooth architectures? Most existing theory only covers networks with one hidden layer, so can we go deeper? In this paper, we focus on recurrent neural networks (RNNs) which are multi-layer networks widely used in natural language processing. They are harder to analyze than feedforward neural networks, because the textitsame recurrent unit is repeatedly applied across the entire time horizon of length L, which is analogous to feedforward networks of depth L. We show when the number of neurons is sufficiently large, meaning polynomial in the training data size and in L, then SGD is capable of minimizing the regression loss in the linear convergence rate. This gives theoretical evidence of how RNNs can memorize data. More importantly, in this paper we build general toolkits to analyze multi-layer networks with ReLU activations. For instance, we prove why ReLU activations can prevent exponential gradient explosion or vanishing, and build a perturbation theory to analyze first-order approximation of multi-layer networks.

Key facts. Cited by 82 papers in the map (data through 14 Oct 2026). First posted on arXiv 29 Oct 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 25; 2022: 16; 2023: 24; 2024: 10; 2025: 3; 2026: 4.

82 papers.

Sort:

By area:

  1. 2021Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learningcited by 119
  2. 2021Learning with invariances in random features and kernel modelscited by 89
  3. 2021Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Modelscited by 76
  4. 2021Training Multi-Layer Over-Parametrized Neural Network in Subquadratic Timecited by 62
  5. 2021Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularizationcited by 49
  6. 2023Attention Scheme Inspired Softmax Regressioncited by 49
  7. 2024LoRA Training in the NTK Regime has No Spurious Local Minimacited by 48
  8. 2022Spectrum of inner-product kernel matrices in the polynomial regime and multiple descent phenomenon in kernel ridge regressioncited by 45
  9. 2023An Over-parameterized Exponential Regressioncited by 45
  10. 2022Sub-Task Decomposition Enables Learning in Sequence to Sequence Taskscited by 41
  11. 2022The Mechanism of Prediction Head in Non-contrastive Self-supervised Learningcited by 38
  12. 2023An Iterative Algorithm for Rescaled Hyperbolic Functions Regressioncited by 35
38 more
  1. 2021Fast Sketching of Polynomial Kernels of Polynomial Degreecited by 33
  2. 2021Does Preprocessing Help Training Over-parameterized Neural Networks?cited by 32
  3. 2021High Quality Segmentation for Ultra High-resolution Imagescited by 31
  4. 2023On the Eigenvalue Decay Rates of a Class of Neural-Network Related Kernel Functions Defined on General Domainscited by 31
  5. 2023InfoPrompt: Information-Theoretic Soft Prompt Tuning for Natural Language Understandingcited by 30
  6. 2023A Fast Optimization View: Reformulating Single Layer Attention in LLM Based on Tensor and SVM Trick, and Solving It in Matrix Multiplication Timecited by 30
  7. 2022Bypass Exponential Time Preprocessing: Fast Neural Network Training via Weight-Data Correlation Preprocessingcited by 29
  8. 2021Scaling Laws for Deep Learningcited by 26
  9. 2023How to Protect Copyright Data in Optimization of Large Language Models?cited by 26
  10. 2024Evaluating the design space of diffusion-based generative modelscited by 26
  11. 2021SGD: The Role of Implicit Regularization, Batch-size and Multiple-epochscited by 22
  12. 2022Training Overparametrized Neural Networks in Sublinear Timecited by 22
  13. 2022A Sublinear Adversarial Training Algorithmcited by 22
  14. 2021Learning with convolution and pooling operations in kernel methodscited by 21
  15. 2023Six Lectures on Linearized Neural Networkscited by 20
  16. 2021Communication-efficient SGD: From Local SGD to One-Shot Averagingcited by 19
  17. 2021Near-optimal Offline and Streaming Algorithms for Learning Non-Linear Dynamical Systemscited by 17
  18. 2022Federated Adversarial Learning: A Framework with Convergence Analysiscited by 17
  19. 2025LoR-VP: Low-Rank Visual Prompting for Efficient Vision Model Adaptationcited by 17
  20. 2022An Analysis of Attention via the Lens of Exchangeability and Latent Variable Modelscited by 16
  21. 2022Bounding the Width of Neural Networks via Coupled Initialization -- A Worst Case Analysiscited by 15
  22. 2023A Brief Survey on the Approximation Theory for Sequence Modellingcited by 15
  23. 2023Why Softmax Attention Outperforms Linear Attentioncited by 15
  24. 2021The Discovery of Dynamics via Linear Multistep Methods and Deep Learning: Error Estimationcited by 14
  25. 2024Differential Privacy Mechanisms in Neural Tangent Kernel Regressioncited by 14
  26. 2021FL-NTK: A Neural Tangent Kernel-based Framework for Federated Learning Convergence Analysiscited by 13
  27. 2023On the Convergence of Encoder-only Shallow Transformerscited by 13
  28. 2024HiLo: Detailed and Robust 3D Clothed Human Reconstruction with High-and Low-Frequency Information of Parametric Modelscited by 13
  29. 2023Efficient SGD Neural Network Training via Sublinear Activated Neuron Identificationcited by 11
  30. 2021Quantifying Epistemic Uncertainty in Deep Learningcited by 10
  31. 2022Global Convergence of Over-parameterized Deep Equilibrium Modelscited by 10
  32. 2023A Scalable Walsh-Hadamard Regularizer to Overcome the Low-degree Spectral Bias of Neural Networkscited by 10
  33. 2023Efficient Uncertainty Quantification and Reduction for Over-Parameterized Neural Networkscited by 10
  34. 2023How many Neurons do we need? A refined Analysis for Shallow Networks trained with Gradient Descentcited by 10
  35. 2022Implicit Bias of MSE Gradient Optimization in Underparameterized Neural Networkscited by 9
  36. 2023An Automatic Learning Rate Schedule Algorithm for Achieving Faster Convergence and Steeper Descentcited by 9
  37. 2021Why Do Local Methods Solve Nonconvex Problems?cited by 8
  38. 2023Is Solving Graph Neural Tangent Kernel Equivalent to Training Graph Neural Network?cited by 8

50 most cited of 82 All 82 (JSON)