# Recurrent Neural Networks and Long Short-Term Memory Networks: Tutorial and Survey

arXiv: https://arxiv.org/abs/2304.11461 · HTML: https://mlsanity.com/paper/2304.11461/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 59 references that are in the map.
- First posted on arXiv 22 Apr 2023.

## Builds on (most cited first, 18 of 18)

- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Deep Learning in Neural Networks: An Overview](https://mlsanity.com/paper/1404.7828/) (2014)
- [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013)
- [Generating Sequences With Recurrent Neural Networks](https://mlsanity.com/paper/1308.0850/) (2013)
- [Bidirectional LSTM-CRF Models for Sequence Tagging](https://mlsanity.com/paper/1508.01991/) (2015)
- [LSTM: A Search Space Odyssey](https://mlsanity.com/paper/1503.04069/) (2015)
- [Unitary Evolution Recurrent Neural Networks](https://mlsanity.com/paper/1511.06464/) (2015)
- [A Critical Review of Recurrent Neural Networks for Sequence Learning](https://mlsanity.com/paper/1506.00019/) (2015)
- [Understanding LSTM -- a tutorial into Long Short-Term Memory Recurrent Neural Networks](https://mlsanity.com/paper/1909.09586/) (2019)
- [Recent Advances in Recurrent Neural Networks](https://mlsanity.com/paper/1801.01078/) (2017)
- [Semi-supervised sequence tagging with bidirectional language models](https://mlsanity.com/paper/1705.00108/) (2017)
- [Eigenvalue and Generalized Eigenvalue Problems: Tutorial](https://mlsanity.com/paper/1903.11240/) (2019)
- [KKT Conditions, First-Order and Second-Order Optimization, and Distributed Optimization: Tutorial and Survey](https://mlsanity.com/paper/2110.01858/) (2021)
- [Simplified Minimal Gated Unit Variations for Recurrent Neural Networks](https://mlsanity.com/paper/1701.03452/) (2017)
- [Learning Longer Memory in Recurrent Neural Networks](https://mlsanity.com/paper/1412.7753/) (2014)
- [Overcoming the vanishing gradient problem in plain recurrent networks](https://mlsanity.com/paper/1801.06105/) (2018)

## Built on by (most cited first, 10 of 10)

- [Human-inspired Perspectives: A Survey on AI Long-term Memory](https://mlsanity.com/paper/2411.00489/) (2024)
- [Recurrent Stochastic Configuration Networks for Temporal Data Analytics](https://mlsanity.com/paper/2406.16959/) (2024)
- [Physics-based Reduced Order Modeling for Uncertainty Quantification of Guided Wave Propagation using Bayesian Optimization](https://mlsanity.com/paper/2307.09661/) (2023)
- [Predicting Post-Traumatic Epilepsy from Clinical Records using Large Language Model Embeddings](https://mlsanity.com/paper/2604.14547/) (2026)
- [Recurrent Stochastic Configuration Networks with Hybrid Regularization for Nonlinear Dynamics Modelling](https://mlsanity.com/paper/2412.00070/) (2024)
- [Can Modern NLP Systems Reliably Annotate Chest Radiography Exams? A Pre-Purchase Evaluation and Comparative Study of Solutions from AWS, Google, Azure, John Snow Labs, and Open-Source Models on an Independent Pediatric Dataset](https://mlsanity.com/paper/2505.23030/) (2025)
- [CNN-LSTM Hybrid Model for AI-Driven Prediction of COVID-19 Severity from Spike Sequences and Clinical Data](https://mlsanity.com/paper/2505.23879/) (2025)
- [Topological Neural Dynamics: A Neuron-wise Framework for Sequence Modeling](https://mlsanity.com/paper/2606.21295/) (2026)
- [Lightweight Safe Reinforcement Learning for End-to-End UAV Navigation](https://mlsanity.com/paper/2607.01794/) (2026)
- [MTLiquid: Enabling Efficient Multi-Task Learning using Liquid Neural Networks for Lightweight Healthcare Monitoring Systems](https://mlsanity.com/paper/2609.33232/) (2026)

## Abstract (arXiv)

This is a tutorial paper on Recurrent Neural Network (RNN), Long Short-Term Memory Network (LSTM), and their variants. We start with a dynamical system and backpropagation through time for RNN. Then, we discuss the problems of gradient vanishing and explosion in long-term dependencies. We explain close-to-identity weight matrix, long delays, leaky units, and echo state networks for solving this problem. Then, we introduce LSTM gates and cells, history and variants of LSTM, and Gated Recurrent Units (GRU). Finally, we introduce bidirectional RNN, bidirectional LSTM, and the Embeddings from Language Model (ELMo) network, for processing a sequence in both directions.