# Towards Non-saturating Recurrent Units for Modelling Long-term Dependencies

arXiv: https://arxiv.org/abs/1902.06704 · HTML: https://mlsanity.com/paper/1902.06704/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Jan 2019.

## Built on by (most cited first, 15 of 15)

- [RWKV: Reinventing RNNs for the Transformer Era](https://mlsanity.com/paper/2305.13048/) (2023)
- [Deep networks for system identification: a Survey](https://mlsanity.com/paper/2301.12832/) (2023)
- [Parallelizing Legendre Memory Unit Training](https://mlsanity.com/paper/2102.11417/) (2021)
- [Exploiting Multiple Timescales in Hierarchical Echo State Networks](https://mlsanity.com/paper/2101.04223/) (2021)
- [How Does Momentum Benefit Deep Neural Networks Architecture Design? A Few Case Studies](https://mlsanity.com/paper/2110.07034/) (2021)
- [Slower is Better: Revisiting the Forgetting Mechanism in LSTM for Slower Information Decay](https://mlsanity.com/paper/2105.05944/) (2021)
- [Random orthogonal additive filters: a solution to the vanishing/exploding gradient of deep neural networks](https://mlsanity.com/paper/2210.01245/) (2022)
- [Delayed Memory Unit: Modelling Temporal Dependency Through Delay Gate](https://mlsanity.com/paper/2310.14982/) (2023)
- [Discrete Function Bases and Convolutional Neural Networks](https://mlsanity.com/paper/2103.05609/) (2021)
- [Investigating Action Encodings in Recurrent Neural Networks in Reinforcement Learning](https://mlsanity.com/paper/2605.16318/) (2026)
- [Fast Saturating Gate for Learning Long Time Scales with Recurrent Neural Networks](https://mlsanity.com/paper/2210.01348/) (2022)
- [Learning Transductions and Alignments with RNN Seq2seq Models](https://mlsanity.com/paper/2303.06841/) (2023)
- [Extending Memory for Language Modelling](https://mlsanity.com/paper/2305.11462/) (2023)
- [Short-Term Load Forecasting Using A Particle-Swarm Optimized Multi-Head Attention-Augmented CNN-LSTM Network](https://mlsanity.com/paper/2309.03694/) (2023)
- [Greedy dynamical meta-learning](https://mlsanity.com/paper/2607.23925/) (2026)

## Abstract (arXiv)

Modelling long-term dependencies is a challenge for recurrent neural networks. This is primarily due to the fact that gradients vanish during training, as the sequence length increases. Gradients can be attenuated by transition operators and are attenuated or dropped by activation functions. Canonical architectures like LSTM alleviate this issue by skipping information through a memory mechanism. We propose a new recurrent architecture (Non-saturating Recurrent Unit; NRU) that relies on a memory mechanism but forgoes both saturating activation functions and saturating gates, in order to further alleviate vanishing gradients. In a series of synthetic and real world tasks, we demonstrate that the proposed model is the only model that performs among the top 2 models across all tasks with and without long-term dependencies, when compared against a range of other architectures.