# Overcoming the vanishing gradient problem in plain recurrent networks

arXiv: https://arxiv.org/abs/1801.06105 · HTML: https://mlsanity.com/paper/1801.06105/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jan 2018.

## Built on by (most cited first, 10 of 10)

- [Conformal prediction for text infilling and part-of-speech prediction](https://mlsanity.com/paper/2111.02592/) (2021)
- [Recurrent Neural Networks and Long Short-Term Memory Networks: Tutorial and Survey](https://mlsanity.com/paper/2304.11461/) (2023)
- [A Survey of Source Code Representations for Machine Learning-Based Cybersecurity Tasks](https://mlsanity.com/paper/2403.10646/) (2024)
- [Ensemble perspective for understanding temporal credit assignment](https://mlsanity.com/paper/2102.03740/) (2021)
- [Attention based Sequence to Sequence Learning for Machine Translation of Low Resourced Indic Languages -- A case of Sanskrit to Hindi](https://mlsanity.com/paper/2110.00435/) (2021)
- [WATUNet: A Deep Neural Network for Segmentation of Volumetric Sweep Imaging Ultrasound](https://mlsanity.com/paper/2311.10857/) (2023)
- [Activation Function Optimization Scheme for Image Classification](https://mlsanity.com/paper/2409.04915/) (2024)
- [Improving the Performance and Learning Stability of Parallelizable RNNs Designed for Ultra-Low Power Applications](https://mlsanity.com/paper/2605.11855/) (2026)
- [A Comprehensive Review on Summarizing Financial News Using Deep Learning](https://mlsanity.com/paper/2109.10118/) (2021)
- [Hybrid of DiffStride and Spectral Pooling in Convolutional Neural Networks](https://mlsanity.com/paper/2401.09008/) (2024)

## Abstract (arXiv)

Plain recurrent networks greatly suffer from the vanishing gradient problem while Gated Neural Networks (GNNs) such as Long-short Term Memory (LSTM) and Gated Recurrent Unit (GRU) deliver promising results in many sequence learning tasks through sophisticated network designs. This paper shows how we can address this problem in a plain recurrent network by analyzing the gating mechanisms in GNNs. We propose a novel network called the Recurrent Identity Network (RIN) which allows a plain recurrent network to overcome the vanishing gradient problem while training very deep models without the use of gates. We compare this model with IRNNs and LSTMs on multiple sequence modeling benchmarks. The RINs demonstrate competitive performance and converge faster in all tasks. Notably, small RIN models produce 12%--67% higher accuracy on the Sequential and Permuted MNIST datasets and reach state-of-the-art performance on the bAbI question answering dataset.