# Do RNN and LSTM have Long Memory?

arXiv: https://arxiv.org/abs/2006.03860 · HTML: https://mlsanity.com/paper/2006.03860/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Jun 2020.

## Built on by (most cited first, 30 of 47)

- [CKConv: Continuous Kernel Convolution For Sequential Data](https://mlsanity.com/paper/2102.02611/) (2021)
- [Scene-Intuitive Agent for Remote Embodied Visual Grounding](https://mlsanity.com/paper/2103.12944/) (2021)
- [Noisy Recurrent Neural Networks](https://mlsanity.com/paper/2102.04877/) (2021)
- [LLMR: Real-time Prompting of Interactive Worlds using Large Language Models](https://mlsanity.com/paper/2309.12276/) (2023)
- [Unlocking the Power of LSTM for Long Term Time Series Forecasting](https://mlsanity.com/paper/2408.10006/) (2024)
- [A Multi-view Multi-task Learning Framework for Multi-variate Time Series Forecasting](https://mlsanity.com/paper/2109.01657/) (2021)
- [A Temporal-Pattern Backdoor Attack to Deep Reinforcement Learning](https://mlsanity.com/paper/2205.02589/) (2022)
- [Recurrence along Depth: Deep Convolutional Neural Networks with Recurrent Layer Aggregation](https://mlsanity.com/paper/2110.11852/) (2021)
- [Enhancing Automatic Modulation Recognition through Robust Global Feature Extraction](https://mlsanity.com/paper/2401.01056/) (2024)
- [On the Memory Mechanism of Tensor-Power Recurrent Models](https://mlsanity.com/paper/2103.01521/) (2021)
- [A Note on Learning Rare Events in Molecular Dynamics using LSTM and Transformer](https://mlsanity.com/paper/2107.06573/) (2021)
- [Recurrent Neural Networks for Learning Long-term Temporal Dependencies with Reanalysis of Time Scale Representation](https://mlsanity.com/paper/2111.03282/) (2021)
- [Xtal2DoS: Attention-based Crystal to Sequence Learning for Density of States Prediction](https://mlsanity.com/paper/2302.01486/) (2023)
- [Deep Generative Spatiotemporal Engression for Probabilistic Forecasting of Epidemics](https://mlsanity.com/paper/2603.07108/) (2026)
- [Recurrence-in-Recurrence Networks for Video Deblurring](https://mlsanity.com/paper/2203.06418/) (2022)
- [Impartial Games: A Challenge for Reinforcement Learning](https://mlsanity.com/paper/2205.12787/) (2022)
- [Don't Watch Me: A Spatio-Temporal Trojan Attack on Deep-Reinforcement-Learning-Augment Autonomous Driving](https://mlsanity.com/paper/2211.14440/) (2022)
- [GAT-GAN : A Graph-Attention-based Time-Series Generative Adversarial Network](https://mlsanity.com/paper/2306.01999/) (2023)
- [UniFault: A Fault Diagnosis Foundation Model from Bearing Data](https://mlsanity.com/paper/2504.01373/) (2025)
- [Anamnesic Neural Differential Equations with Orthogonal Polynomial Projections](https://mlsanity.com/paper/2303.01841/) (2023)
- [A Generalized Transformer-based Radio Link Failure Prediction Framework in 5G RANs](https://mlsanity.com/paper/2407.05197/) (2024)
- [A Foundation Model for Wearable Movement Data in Mental Health Research](https://mlsanity.com/paper/2411.15240/) (2024)
- [IMSSA: Deploying modern state-space models on memristive in-memory compute hardware](https://mlsanity.com/paper/2412.20215/) (2024)
- [Deep Inertial Odometry with Accurate IMU Preintegration](https://mlsanity.com/paper/2101.07061/) (2021)
- [ARISE: ApeRIodic SEmi-parametric Process for Efficient Markets without Periodogram and Gaussianity Assumptions](https://mlsanity.com/paper/2111.06222/) (2021)
- [TKN: Transformer-based Keypoint Prediction Network For Real-time Video Prediction](https://mlsanity.com/paper/2303.09807/) (2023)
- [Improve Long-term Memory Learning Through Rescaling the Error Temporally](https://mlsanity.com/paper/2307.11462/) (2023)
- [Learning Successor Features with Distributed Hebbian Temporal Memory](https://mlsanity.com/paper/2310.13391/) (2023)
- [Exploring Parity Challenges in Reinforcement Learning through Curriculum Learning with Noisy Labels](https://mlsanity.com/paper/2312.05379/) (2023)
- [Curriculum Design Helps Spiking Neural Networks to Classify Time Series](https://mlsanity.com/paper/2401.10257/) (2023)

## Abstract (arXiv)

The LSTM network was proposed to overcome the difficulty in learning long-term dependence, and has made significant advancements in applications. With its success and drawbacks in mind, this paper raises the question - do RNN and LSTM have long memory? We answer it partially by proving that RNN and LSTM do not have long memory from a statistical perspective. A new definition for long memory networks is further introduced, and it requires the model weights to decay at a polynomial rate. To verify our theory, we convert RNN and LSTM into long memory networks by making a minimal modification, and their superiority is illustrated in modeling long-term dependence of various datasets.