# Quantifying the vanishing gradient and long distance dependency problem in recursive neural networks and recursive LSTMs

arXiv: https://arxiv.org/abs/1603.00423 · HTML: https://mlsanity.com/paper/1603.00423/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Mar 2016.

## Built on by (most cited first, 21 of 21)

- [RWKV: Reinventing RNNs for the Transformer Era](https://mlsanity.com/paper/2305.13048/) (2023)
- [Dual-path Mamba: Short and Long-term Bidirectional Selective Structured State Space Models for Speech Separation](https://mlsanity.com/paper/2403.18257/) (2024)
- [Creativity and Machine Learning: A Survey](https://mlsanity.com/paper/2104.02726/) (2021)
- [Learning from Predictions: Fusing Training and Autoregressive Inference for Long-Term Spatiotemporal Forecasts](https://mlsanity.com/paper/2302.11101/) (2023)
- [Long Short-term Memory RNN](https://mlsanity.com/paper/2105.06756/) (2021)
- [An Empirical Study: Extensive Deep Temporal Point Process](https://mlsanity.com/paper/2110.09823/) (2021)
- [Predictive Maintenance for General Aviation Using Convolutional Transformers](https://mlsanity.com/paper/2110.03757/) (2021)
- [Neural Spatiotemporal Point Processes: Trends and Challenges](https://mlsanity.com/paper/2502.09341/) (2025)
- [Robust Augmentation for Multivariate Time Series Classification](https://mlsanity.com/paper/2201.11739/) (2022)
- [Time Series Forecasting (TSF) Using Various Deep Learning Models](https://mlsanity.com/paper/2204.11115/) (2022)
- [A Large-Scale Annotated Multivariate Time Series Aviation Maintenance Dataset from the NGAFID](https://mlsanity.com/paper/2210.07317/) (2022)
- [RRWKV: Capturing Long-range Dependencies in RWKV](https://mlsanity.com/paper/2306.05176/) (2023)
- [Enhancing Transformer RNNs with Multiple Temporal Perspectives](https://mlsanity.com/paper/2402.02625/) (2024)
- [A Foundation Model for Wearable Movement Data in Mental Health Research](https://mlsanity.com/paper/2411.15240/) (2024)
- [Advances in Transformers for Robotic Applications: A Review](https://mlsanity.com/paper/2412.10599/) (2024)
- [Real-time Forecasting of Time Series in Financial Markets Using Sequentially Trained Many-to-one LSTMs](https://mlsanity.com/paper/2205.04678/) (2022)
- [Machine Learning based prediction of Glucose Levels in Type 1 Diabetes Patients with the use of Continuous Glucose Monitoring Data](https://mlsanity.com/paper/2302.12856/) (2023)
- [Human-mediated Large Language Models for Robotic Intervention in Children with Autism Spectrum Disorders](https://mlsanity.com/paper/2402.00260/) (2024)
- [The Interpretability of LSTM Models for Predicting Oil Company Stocks: Impact of Correlated Features](https://mlsanity.com/paper/2201.00350/) (2022)
- [Experimental Study on Time Series Analysis of Lower Limb Rehabilitation Exercise Data Driven by Novel Model Architecture and Large Models](https://mlsanity.com/paper/2504.03799/) (2025)
- [Predicting Grain Growth in Polycrystalline Materials Using Deep Learning Time Series Models](https://mlsanity.com/paper/2511.11630/) (2025)

## Abstract (arXiv)

Recursive neural networks (RNN) and their recently proposed extension recursive long short term memory networks (RLSTM) are models that compute representations for sentences, by recursively combining word embeddings according to an externally provided parse tree. Both models thus, unlike recurrent networks, explicitly make use of the hierarchical structure of a sentence. In this paper, we demonstrate that RNNs nevertheless suffer from the vanishing gradient and long distance dependency problem, and that RLSTMs greatly improve over RNN's on these problems. We present an artificial learning task that allows us to quantify the severity of these problems for both models. We further show that a ratio of gradients (at the root node and a focal leaf node) is highly indicative of the success of backpropagation at optimizing the relevant weights low in the tree. This paper thus provides an explanation for existing, superior results of RLSTMs on tasks such as sentiment analysis, and suggests that the benefits of including hierarchical structure and of including LSTM-style gating are complementary.