Adaptive-saturated RNN: Remember more with less instability
Not on the map. Why?
Abstract (arXiv)
Orthogonal parameterization is a compelling solution to the vanishing gradient problem (VGP) in recurrent neural networks (RNNs). With orthogonal parameters and non-saturated activation functions, gradients in such models are constrained to unit norms. On the other hand, although the traditional vanilla RNNs are seen to have higher memory capacity, they suffer from the VGP and perform badly in many applications. This work proposes Adaptive-Saturated RNNs (asRNN), a variant that dynamically adjusts its saturation level between the two mentioned approaches. Consequently, asRNN enjoys both the capacity of a vanilla RNN and the training stability of orthogonal RNNs. Our experiments show encouraging results of asRNN on challenging sequence learning benchmarks compared to several strong competitors. The research code is accessible at https://github.com/ndminhkhoi46/asRNN/.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 9 of its 16 references that are in the map. First posted on arXiv 24 Apr 2023.
Builds on
9 of 16 references.
References · 9
- 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 722
- 2015Unitary Evolution Recurrent Neural Networkscited by 294
- 2019Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Groupcited by 134
- 2017Orthogonal Recurrent Neural Networks with Scaled Cayley Transformcited by 83
- 2016Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNscited by 65
- 2016Recurrent Orthogonal Networks and Long-Memory Taskscited by 38
- 2019Non-normal Recurrent Neural Network (nnRNN): learning long time dependencies while improving expressivity with transient dynamicscited by 29
- 2016Capacity and Trainability in Recurrent Neural Networkscited by 28
- 2015Alternative structures for character-level RNNscited by 5
Built on by
Not cited yet.