# On Generalization Bounds of a Family of Recurrent Neural Networks

arXiv: https://arxiv.org/abs/1910.12947 · HTML: https://mlsanity.com/paper/1910.12947/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Oct 2019.

## Built on by (most cited first, 30 of 42)

- [Score Approximation, Estimation and Distribution Recovery of Diffusion Models on Low-Dimensional Data](https://mlsanity.com/paper/2302.07194/) (2023)
- [Inductive Biases and Variable Creation in Self-Attention Mechanisms](https://mlsanity.com/paper/2110.10090/) (2021)
- [Theory of Graph Neural Networks: Representation and Learning](https://mlsanity.com/paper/2204.07697/) (2022)
- [Framing RNN as a kernel method: A neural ODE approach](https://mlsanity.com/paper/2106.01202/) (2021)
- [Bounding the Rademacher Complexity of Fourier neural operators](https://mlsanity.com/paper/2209.05150/) (2022)
- [A Brief Survey on the Approximation Theory for Sequence Modelling](https://mlsanity.com/paper/2302.13752/) (2023)
- [On Recurrent Neural Networks for learning-based control: recent results and ideas for future developments](https://mlsanity.com/paper/2111.13557/) (2021)
- [From Generalization Analysis to Optimization Designs for State Space Models](https://mlsanity.com/paper/2405.02670/) (2024)
- [Communication-Efficient Federated Learning with Dual-Side Low-Rank Compression](https://mlsanity.com/paper/2104.12416/) (2021)
- [Provable Benefits of Complex Parameterizations for Structured State Space Models](https://mlsanity.com/paper/2410.14067/) (2024)
- [Few-Shot Learning by Dimensionality Reduction in Gradient Space](https://mlsanity.com/paper/2206.03483/) (2022)
- [On the Provable Generalization of Recurrent Neural Networks](https://mlsanity.com/paper/2109.14142/) (2021)
- [On Size-Independent Sample Complexity of ReLU Networks](https://mlsanity.com/paper/2306.01992/) (2023)
- [Generalization Error Analysis for Selective State-Space Models Through the Lens of Attention](https://mlsanity.com/paper/2502.01473/) (2025)
- [The Panaceas for Improving Low-Rank Decomposition in Communication-Efficient Federated Learning](https://mlsanity.com/paper/2505.23176/) (2025)
- [On the Importance of Sampling in Training GCNs: Tighter Analysis and Variance Reduction](https://mlsanity.com/paper/2103.02696/) (2021)
- [Cerberus: Exploring Federated Prediction of Security Events](https://mlsanity.com/paper/2209.03050/) (2022)
- [PAC-Bayesian bounds for learning LTI-ss systems with input from empirical loss](https://mlsanity.com/paper/2303.16816/) (2023)
- [Generalization Bounds for Neural Belief Propagation Decoders](https://mlsanity.com/paper/2305.10540/) (2023)
- [PAC-Bayes Generalisation Bounds for Dynamical Systems Including Stable RNNs](https://mlsanity.com/paper/2312.09793/) (2023)
- [Approximation Bounds for Recurrent Neural Networks with Application to Regression](https://mlsanity.com/paper/2409.05577/) (2024)
- [A Generalization Bound for a Family of Implicit Networks](https://mlsanity.com/paper/2410.07427/) (2024)
- [Spectral Pruning for Recurrent Neural Networks](https://mlsanity.com/paper/2105.10832/) (2021)
- [On the Generalization and Approximation Capacities of Neural Controlled Differential Equations](https://mlsanity.com/paper/2305.16791/) (2023)
- [State space models can express n-gram languages](https://mlsanity.com/paper/2306.17184/) (2023)
- [A Distance Correlation-Based Approach to Characterize the Effectiveness of Recurrent Neural Networks for Time Series Forecasting](https://mlsanity.com/paper/2307.15830/) (2023)
- [When Do Transformers Outperform Feedforward and Recurrent Networks? A Statistical Perspective](https://mlsanity.com/paper/2503.11272/) (2025)
- [Length independent generalization bounds for deep SSM architectures via Rademacher contraction and stability constraints](https://mlsanity.com/paper/2405.20278/) (2024)
- [On Non-asymptotic Theory of Recurrent Neural Networks in Temporal Point Processes](https://mlsanity.com/paper/2406.00630/) (2024)
- [Generalization and Risk Bounds for Recurrent Neural Networks](https://mlsanity.com/paper/2411.02784/) (2024)

## Abstract (arXiv)

Recurrent Neural Networks (RNNs) have been widely applied to sequential data analysis. Due to their complicated modeling structures, however, the theory behind is still largely missing. To connect theory and practice, we study the generalization properties of vanilla RNNs as well as their variants, including Minimal Gated Unit (MGU), Long Short Term Memory (LSTM), and Convolutional (Conv) RNNs. Specifically, our theory is established under the PAC-Learning framework. The generalization bound is presented in terms of the spectral norms of the weight matrices and the total number of parameters. We also establish refined generalization bounds with additional norm assumptions, and draw a comparison among these bounds. We remark: (1) Our generalization bound for vanilla RNNs is significantly tighter than the best of existing results; (2) We are not aware of any other generalization bounds for MGU, LSTM, and Conv RNNs in the exiting literature; (3) We demonstrate the advantages of these variants in generalization.