# Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN

arXiv: https://arxiv.org/abs/1803.04831 · HTML: https://mlsanity.com/paper/1803.04831/

## Key facts

- Cited by 143 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Mar 2018.

## Built on by (most cited first, 30 of 143)

- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Simplified State Space Layers for Sequence Modeling](https://mlsanity.com/paper/2208.04933/) (2022)
- [Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers](https://mlsanity.com/paper/2110.13985/) (2021)
- [Resurrecting Recurrent Neural Networks for Long Sequences](https://mlsanity.com/paper/2303.06349/) (2023)
- [Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2107.12213/) (2021)
- [Liquid Structural State-Space Models](https://mlsanity.com/paper/2209.12951/) (2022)
- [CKConv: Continuous Kernel Convolution For Sequential Data](https://mlsanity.com/paper/2102.02611/) (2021)
- [Accurate and efficient time-domain classification with adaptive spiking recurrent neural networks](https://mlsanity.com/paper/2103.12593/) (2021)
- [Skeleton Aware Multi-modal Sign Language Recognition](https://mlsanity.com/paper/2103.08833/) (2021)
- [Video Summarization Using Deep Neural Networks: A Survey](https://mlsanity.com/paper/2101.06072/) (2021)
- [On the difficulty of learning chaotic dynamics with RNNs](https://mlsanity.com/paper/2110.07238/) (2021)
- [FlexConv: Continuous Kernel Convolutions with Differentiable Kernel Sizes](https://mlsanity.com/paper/2110.08059/) (2021)
- [iMiGUE: An Identity-free Video Dataset for Micro-Gesture Understanding and Emotion Analysis](https://mlsanity.com/paper/2107.00285/) (2021)
- [POPGym: Benchmarking Partially Observable Reinforcement Learning](https://mlsanity.com/paper/2303.01859/) (2023)
- [Learning Discriminative Representations for Skeleton Based Action Recognition](https://mlsanity.com/paper/2303.03729/) (2023)
- [UnICORNN: A recurrent model for learning very long time dependencies](https://mlsanity.com/paper/2103.05487/) (2021)
- [SkateFormer: Skeletal-Temporal Transformer for Human Action Recognition](https://mlsanity.com/paper/2403.09508/) (2024)
- [Long Expressive Memory for Sequence Modeling](https://mlsanity.com/paper/2110.04744/) (2021)
- [Machine Learning in Aerodynamic Shape Optimization](https://mlsanity.com/paper/2202.07141/) (2022)
- [IGFormer: Interaction Graph Transformer for Skeleton-based Human Interaction Recognition](https://mlsanity.com/paper/2207.12100/) (2022)
- [Review of Pedestrian Trajectory Prediction Methods: Comparing Deep Learning and Knowledge-based Approaches](https://mlsanity.com/paper/2111.06740/) (2021)
- [Efficient recurrent architectures through activity sparsity and sparse back-propagation through time](https://mlsanity.com/paper/2206.06178/) (2022)
- [Expansion-Squeeze-Excitation Fusion Network for Elderly Activity Recognition](https://mlsanity.com/paper/2112.10992/) (2021)
- [Hulk: A Universal Knowledge Translator for Human-Centric Tasks](https://mlsanity.com/paper/2312.01697/) (2023)
- [AdaSGN: Adapting Joint Number and Model Size for Efficient Skeleton-Based Action Recognition](https://mlsanity.com/paper/2103.11770/) (2021)
- [Is attention to bounding boxes all you need for pedestrian action prediction?](https://mlsanity.com/paper/2107.08031/) (2021)
- [Exploring the Promise and Limits of Real-Time Recurrent Learning](https://mlsanity.com/paper/2305.19044/) (2023)
- [Deep Learning for Android Malware Defenses: a Systematic Literature Review](https://mlsanity.com/paper/2103.05292/) (2021)
- [Heterogeneous Multi-task Learning with Expert Diversity](https://mlsanity.com/paper/2106.10595/) (2021)
- [Focal and Global Spatial-Temporal Transformer for Skeleton-based Action Recognition](https://mlsanity.com/paper/2210.02693/) (2022)

## Abstract (arXiv)

Recurrent neural networks (RNNs) have been widely used for processing sequential data. However, RNNs are commonly difficult to train due to the well-known gradient vanishing and exploding problems and hard to learn long-term patterns. Long short-term memory (LSTM) and gated recurrent unit (GRU) were developed to address these problems, but the use of hyperbolic tangent and the sigmoid action functions results in gradient decay over layers. Consequently, construction of an efficiently trainable deep network is challenging. In addition, all the neurons in an RNN layer are entangled together and their behaviour is hard to interpret. To address these problems, a new type of RNN, referred to as independently recurrent neural network (IndRNN), is proposed in this paper, where neurons in the same layer are independent of each other and they are connected across layers. We have shown that an IndRNN can be easily regulated to prevent the gradient exploding and vanishing problems while allowing the network to learn long-term dependencies. Moreover, an IndRNN can work with non-saturated activation functions such as relu (rectified linear unit) and be still trained robustly. Multiple IndRNNs can be stacked to construct a network that is deeper than the existing RNNs. Experimental results have shown that the proposed IndRNN is able to process very long sequences (over 5000 time steps), can be used to construct very deep networks (21 layers used in the experiment) and still be trained robustly. Better performances have been achieved on various tasks by using IndRNNs compared with the traditional RNN and LSTM. The code is available at https://github.com/Sunnydreamrain/IndRNN_Theano_Lasagne.