Global Optimality of Elman-type RNN in the Mean-Field Regime
Not on the map. Why?
Abstract (arXiv)
We analyze Elman-type Recurrent Reural Networks (RNNs) and their training in the mean-field regime. Specifically, we show convergence of gradient descent training dynamics of the RNN to the corresponding mean-field formulation in the large width limit. We also show that the fixed points of the limiting infinite-width dynamics are globally optimal, under some assumptions on the initialization of the weights. Our results establish optimality for feature-learning with wide RNNs in the mean-field regime
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 20 of its 42 references that are in the map. First posted on arXiv 12 Mar 2023.
Builds on
20 of 42 references.
References · 20
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
- 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,304
- 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
- 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
- 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 268
- 2019Linearized two-layers neural networks in high dimensioncited by 155
- 2020Tensor Programs II: Neural Tangent Kernel for Any Architecturecited by 143
- 2019Limitations of Lazy Training of Two-layers Neural Networkscited by 87
- 2020A Rigorous Framework for the Mean Field Limit of Multilayer Neural Networkscited by 68
- 2020The Recurrent Neural Tangent Kernelcited by 67
- 2020A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depthcited by 56
8 more
- 2019Sparse Optimization on Measures with Over-parameterized Gradient Descentcited by 50
- 2018Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networkscited by 45
- 2019Mean Field Analysis of Deep Neural Networkscited by 42
- 2020On the Convergence of Gradient Descent Training for Two-layer ReLU-networks in the Mean Field Regimecited by 39
- 2020Global optimality of softmax policy gradient with single hidden layer neural networks in the mean-field regimecited by 17
- 2021Global Convergence of Three-layer Neural Networks in the Mean Field Regimecited by 16
- 2019Asymptotics of Reinforcement Learning with Neural Networkscited by 10
- 2019Temporal-difference learning with nonlinear function approximation: lazy training and mean field regimescited by 5
Built on by
Citing papers per year: 2026: 1.
1 paper.