Approximation Bounds for Recurrent Neural Networks with Application to Regression
Not on the map. Why?
Abstract (arXiv)
We study the approximation capacity of deep ReLU recurrent neural networks (RNNs) and explore the convergence properties of nonparametric least squares regression using RNNs. We derive upper bounds on the approximation error of RNNs for H"older smooth functions, in the sense that the output at each time step of an RNN can approximate a H"older function that depends only on past and current information, termed a past-dependent function. This allows a carefully constructed RNN to simultaneously approximate a sequence of past-dependent H"older functions. We apply these approximation results to derive non-asymptotic upper bounds for the prediction error of the empirical risk minimizer in regression problem. Our error bounds achieve minimax optimal rate under both exponentially β-mixing and i.i.d. data assumptions, improving upon existing ones. Our results provide statistical guarantees on the performance of RNNs.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 35 of its 81 references that are in the map. First posted on arXiv 9 Sep 2024.
Builds on
35 of 81 references.
References · 35
- 2017Attention Is All You Needcited by 59,932
- 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
- 2017Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networkscited by 869
- 2016Error bounds for approximations with deep ReLU networkscited by 567
- 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 431
- 2017Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networkscited by 304
- 2017Optimal approximation of piecewise smooth functions using deep ReLU neural networkscited by 176
- 2018Optimal approximation of continuous functions by very deep ReLU networkscited by 165
- 2020Deep Network Approximation for Smooth Functionscited by 163
- 2018Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionalitycited by 160
- 2015End-to-End Attention-based Large Vocabulary Speech Recognitioncited by 148
- 2019Deep Network Approximation Characterized by Number of Neuronscited by 120
23 more
- 2023Hierarchically Gated Recurrent Neural Network for Sequence Modelingcited by 100
- 2021Optimal Approximation Rate of ReLU Networks in terms of Width and Depthcited by 95
- 2019The phase diagram of approximation rates for deep neural networkscited by 86
- 2018A Priori Estimates of the Population Risk for Two-layer Neural Networkscited by 76
- 2019Deep ReLU network approximation of functions on a manifoldcited by 73
- 2019Nonparametric Regression on Low-Dimensional Manifolds using Deep ReLU Networks : Function Approximation and Statistical Recoverycited by 73
- 2020Neural Network Approximation: Three Hidden Layers Are Enoughcited by 63
- 2020Representation formulas and pointwise properties for Barron functionscited by 62
- 2024RNNs are not Transformers (Yet): The Key Bottleneck on In-context Retrievalcited by 62
- 2023SegRNN: Segment Recurrent Neural Network for Long-Term Time Series Forecastingcited by 53
- 2019Deep learning is adaptive to intrinsic dimensionality of model smoothness in anisotropic Besov spacecited by 46
- 2019On Generalization Bounds of a Family of Recurrent Neural Networkscited by 42
- 2021An error analysis of generative adversarial networks for learning distributionscited by 35
- 2022How do noise tails impact on deep ReLU networks?cited by 25
- 2023Nonparametric regression using over-parameterized shallow ReLU neural networkscited by 24
- 2023Optimal rates of approximation by shallow ReLUk neural networks and applications to nonparametric regressioncited by 15
- 2020On the rate of convergence of a deep recurrent neural network estimate in a regression problem with dependent datacited by 14
- 2020Approximation in shift-invariant spaces with deep ReLU neural networkscited by 12
- 2022Minimal Width for Universal Property of Deep RNNcited by 12
- 2023Rates of Approximation by ReLU Shallow Neural Networkscited by 12
- 2023Optimal Approximation of Zonoids and Uniform Approximation by Shallow Neural Networkscited by 11
- 2013Predictive PAC Learning and Process Decompositionscited by 9
- 2023Statistical Spatially Inhomogeneous Diffusion Inferencecited by 2
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.