# Deep Independently Recurrent Neural Network (IndRNN)

arXiv: https://arxiv.org/abs/1910.06251 · HTML: https://mlsanity.com/paper/1910.06251/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Oct 2019.

## Built on by (most cited first, 14 of 14)

- [Long-term Forecasting with TiDE: Time-series Dense Encoder](https://mlsanity.com/paper/2304.08424/) (2023)
- [UnICORNN: A recurrent model for learning very long time dependencies](https://mlsanity.com/paper/2103.05487/) (2021)
- [A Two-stream Neural Network for Pose-based Hand Gesture Recognition](https://mlsanity.com/paper/2101.08926/) (2021)
- [Recurrent Natural Policy Gradient for POMDPs](https://mlsanity.com/paper/2405.18221/) (2024)
- [RIANN -- A Robust Neural Network Outperforms Attitude Estimation Filters](https://mlsanity.com/paper/2104.07391/) (2021)
- [Deep Learning Approaches for Seizure Video Analysis: A Review](https://mlsanity.com/paper/2312.10930/) (2023)
- [DeepSITH: Efficient Learning via Decomposition of What and When Across Time Scales](https://mlsanity.com/paper/2104.04646/) (2021)
- [Random orthogonal additive filters: a solution to the vanishing/exploding gradient of deep neural networks](https://mlsanity.com/paper/2210.01245/) (2022)
- [Transformers in Action Recognition: A Review on Temporal Modeling](https://mlsanity.com/paper/2302.01921/) (2022)
- [Deep Loss Convexification for Learning Iterative Models](https://mlsanity.com/paper/2411.10649/) (2024)
- [Hierarchical Recurrent Adapters for Efficient Multi-Task Adaptation of Large Speech Models](https://mlsanity.com/paper/2403.19709/) (2024)
- [Image Classification using Sequence of Pixels](https://mlsanity.com/paper/2209.11495/) (2022)
- [Unsupervised Spatial-Temporal Feature Enrichment and Fidelity Preservation Network for Skeleton based Action Recognition](https://mlsanity.com/paper/2401.14034/) (2024)
- [A Review of the Long Horizon Forecasting Problem in Time Series Analysis](https://mlsanity.com/paper/2506.12809/) (2025)

## Abstract (arXiv)

Recurrent neural networks (RNNs) are known to be difficult to train due to the gradient vanishing and exploding problems and thus difficult to learn long-term patterns and construct deep networks. To address these problems, this paper proposes a new type of RNNs with the recurrent connection formulated as Hadamard product, referred to as independently recurrent neural network (IndRNN), where neurons in the same layer are independent of each other and connected across layers. Due to the better behaved gradient backpropagation, IndRNN with regulated recurrent weights effectively addresses the gradient vanishing and exploding problems and thus long-term dependencies can be learned. Moreover, an IndRNN can work with non-saturated activation functions such as ReLU (rectified linear unit) and be still trained robustly. Different deeper IndRNN architectures, including the basic stacked IndRNN, residual IndRNN and densely connected IndRNN, have been investigated, all of which can be much deeper than the existing RNNs. Furthermore, IndRNN reduces the computation at each time step and can be over 10 times faster than the commonly used Long short-term memory (LSTM). Experimental results have shown that the proposed IndRNN is able to process very long sequences and construct very deep networks. Better performance has been achieved on various tasks with IndRNNs compared with the traditional RNN, LSTM and the popular Transformer.