mlsanityData to 7 Oct 2026

On Generalization Bounds of a Family of Recurrent Neural Networks

arXiv 1910.12947 ↗First posted 28 Oct 2019cs.LG stat.MLcited by 42

Not on the map. Why?

Abstract (arXiv)

Recurrent Neural Networks (RNNs) have been widely applied to sequential data analysis. Due to their complicated modeling structures, however, the theory behind is still largely missing. To connect theory and practice, we study the generalization properties of vanilla RNNs as well as their variants, including Minimal Gated Unit (MGU), Long Short Term Memory (LSTM), and Convolutional (Conv) RNNs. Specifically, our theory is established under the PAC-Learning framework. The generalization bound is presented in terms of the spectral norms of the weight matrices and the total number of parameters. We also establish refined generalization bounds with additional norm assumptions, and draw a comparison among these bounds. We remark: (1) Our generalization bound for vanilla RNNs is significantly tighter than the best of existing results; (2) We are not aware of any other generalization bounds for MGU, LSTM, and Conv RNNs in the exiting literature; (3) We demonstrate the advantages of these variants in generalization.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). First posted on arXiv 28 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 6; 2023: 10; 2024: 9; 2025: 5; 2026: 4.

42 papers.

Sort:

By area:

  1. 2023Score Approximation, Estimation and Distribution Recovery of Diffusion Models on Low-Dimensional Datacited by 171
  2. 2021Inductive Biases and Variable Creation in Self-Attention Mechanismscited by 144
  3. 2022Theory of Graph Neural Networks: Representation and Learningcited by 60
  4. 2021Framing RNN as a kernel method: A neural ODE approachcited by 23
  5. 2022Bounding the Rademacher Complexity of Fourier neural operatorscited by 16
  6. 2023A Brief Survey on the Approximation Theory for Sequence Modellingcited by 15
  7. 2021On Recurrent Neural Networks for learning-based control: recent results and ideas for future developmentscited by 14
  8. 2024From Generalization Analysis to Optimization Designs for State Space Modelscited by 11
  9. 2021Communication-Efficient Federated Learning with Dual-Side Low-Rank Compressioncited by 10
  10. 2024Provable Benefits of Complex Parameterizations for Structured State Space Modelscited by 9
  11. 2022Few-Shot Learning by Dimensionality Reduction in Gradient Spacecited by 8
  12. 2021On the Provable Generalization of Recurrent Neural Networkscited by 6
30 more
  1. 2023On Size-Independent Sample Complexity of ReLU Networkscited by 6
  2. 2025Generalization Error Analysis for Selective State-Space Models Through the Lens of Attentioncited by 5
  3. 2025The Panaceas for Improving Low-Rank Decomposition in Communication-Efficient Federated Learningcited by 5
  4. 2021On the Importance of Sampling in Training GCNs: Tighter Analysis and Variance Reductioncited by 4
  5. 2022Cerberus: Exploring Federated Prediction of Security Eventscited by 4
  6. 2023Generalization Bounds for Neural Belief Propagation Decoderscited by 4
  7. 2023PAC-Bayes Generalisation Bounds for Dynamical Systems Including Stable RNNscited by 4
  8. 2024A Generalization Bound for a Family of Implicit Networkscited by 4
  9. 2021Spectral Pruning for Recurrent Neural Networkscited by 3
  10. 2023PAC-Bayesian bounds for learning LTI-ss systems with input from empirical losscited by 3
  11. 2023On the Generalization and Approximation Capacities of Neural Controlled Differential Equationscited by 3
  12. 2023State space models can express n-gram languagescited by 3
  13. 2023A Distance Correlation-Based Approach to Characterize the Effectiveness of Recurrent Neural Networks for Time Series Forecastingcited by 3
  14. 2024Length independent generalization bounds for deep SSM architectures via Rademacher contraction and stability constraintscited by 2
  15. 2024On Non-asymptotic Theory of Recurrent Neural Networks in Temporal Point Processescited by 2
  16. 2024Approximation Bounds for Recurrent Neural Networks with Application to Regressioncited by 2
  17. 2024Generalization and Risk Bounds for Recurrent Neural Networkscited by 2
  18. 2025When Do Transformers Outperform Feedforward and Recurrent Networks? A Statistical Perspectivecited by 2
  19. 2025FlowPath: Learning Data-Driven Manifolds with Invertible Flows for Robust Irregularly-sampled Time Series Classificationcited by 2
  20. 2022PAC-Bayesian-Like Error Bound for a Class of Linear Time-Invariant Stochastic State-Space Modelscited by 1
  21. 2024Universality of reservoir systems with recurrent neural networkscited by 1
  22. 2026Generalization Bounds for Transformer Channel Decoderscited by 1
  23. 2026Cold-Start Forecasting of New Product Life-Cycles via Conditional Diffusion Modelscited by 1
  24. 2021Learning and Generalization in RNNscited by 0
  25. 2022Optimal Solutions for Joint Beamforming and Antenna Selection: From Branch and Bound to Graph Neural Imitation Learningcited by 0
  26. 2023On the Role of Noise in the Sample Complexity of Learning Recurrent Neural Networks: Exponential Gaps for Long Sequencescited by 0
  27. 2024A finite-sample generalization bound for stable LPV systemscited by 0
  28. 2025Learning Decentralized Swarms Using Rotation Equivariant Graph Neural Networkscited by 0
  29. 2026Plug-In Classification of Drift Functions in Diffusion Processes Using Neural Networkscited by 0
  30. 2026Upper Generalization Bounds for Neural Oscillatorscited by 0