# A Critical Review of Recurrent Neural Networks for Sequence Learning

arXiv: https://arxiv.org/abs/1506.00019 · HTML: https://mlsanity.com/paper/1506.00019/

## Key facts

- Cited by 224 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 May 2015.

## Built on by (most cited first, 30 of 224)

- [ChatGPT: Jack of all trades, master of none](https://mlsanity.com/paper/2302.10724/) (2023)
- [Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Survey](https://mlsanity.com/paper/2105.04387/) (2021)
- [A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges](https://mlsanity.com/paper/2406.11903/) (2024)
- [OadTR: Online Action Detection with Transformers](https://mlsanity.com/paper/2106.11149/) (2021)
- [Knowledge Tracing: A Survey](https://mlsanity.com/paper/2201.06953/) (2022)
- [Graph-Based Deep Learning for Medical Diagnosis and Analysis: Past, Present and Future](https://mlsanity.com/paper/2105.13137/) (2021)
- [On the difficulty of learning chaotic dynamics with RNNs](https://mlsanity.com/paper/2110.07238/) (2021)
- [SegRNN: Segment Recurrent Neural Network for Long-Term Time Series Forecasting](https://mlsanity.com/paper/2308.11200/) (2023)
- [Deep Gait Recognition: A Survey](https://mlsanity.com/paper/2102.09546/) (2021)
- [Transformers in Time-series Analysis: A Tutorial](https://mlsanity.com/paper/2205.01138/) (2022)
- [Predicting Loose-Fitting Garment Deformations Using Bone-Driven Motion Networks](https://mlsanity.com/paper/2205.01355/) (2022)
- [Content and Style Aware Generation of Text-line Images for Handwriting Recognition](https://mlsanity.com/paper/2204.05539/) (2022)
- [SenseFi: A Library and Benchmark on Deep-Learning-Empowered WiFi Human Sensing](https://mlsanity.com/paper/2207.07859/) (2022)
- [Click-Through Rate Prediction in Online Advertising: A Literature Review](https://mlsanity.com/paper/2202.10462/) (2022)
- [Core Challenges in Embodied Vision-Language Planning](https://mlsanity.com/paper/2106.13948/) (2021)
- [Vision Transformers for Action Recognition: A Survey](https://mlsanity.com/paper/2209.05700/) (2022)
- [Bringing Generative AI to Adaptive Learning in Education](https://mlsanity.com/paper/2402.14601/) (2024)
- [Anticipating synchronization with machine learning](https://mlsanity.com/paper/2103.13358/) (2021)
- [An Overview and Experimental Study of Learning-based Optimization Algorithms for Vehicle Routing Problem](https://mlsanity.com/paper/2107.07076/) (2021)
- [Machine Learning Methods in Solving the Boolean Satisfiability Problem](https://mlsanity.com/paper/2203.04755/) (2022)
- [Prompt-Matched Semantic Segmentation](https://mlsanity.com/paper/2208.10159/) (2022)
- [BookGPT: A General Framework for Book Recommendation Empowered by Large Language Model](https://mlsanity.com/paper/2305.15673/) (2023)
- [FreeStyle: Free Lunch for Text-guided Style Transfer using Diffusion Models](https://mlsanity.com/paper/2401.15636/) (2024)
- [Learning Long Sequences in Spiking Neural Networks](https://mlsanity.com/paper/2401.00955/) (2023)
- [SpikingSSMs: Learning Long Sequences with Sparse and Parallel Spiking State Space Models](https://mlsanity.com/paper/2408.14909/) (2024)
- [A Survey on Aspect-Based Sentiment Classification](https://mlsanity.com/paper/2203.14266/) (2022)
- [Thermodynamically Consistent Machine-Learned Internal State Variable Approach for Data-Driven Modeling of Path-Dependent Materials](https://mlsanity.com/paper/2205.00578/) (2022)
- [A Survey on Deep Reinforcement Learning for Audio-Based Applications](https://mlsanity.com/paper/2101.00240/) (2021)
- [On the Resurgence of Recurrent Models for Long Sequences -- Survey and Research Opportunities in the Transformer Era](https://mlsanity.com/paper/2402.08132/) (2024)
- [A Framework for Demonstrating Practical Quantum Advantage: Racing Quantum against Classical Generative Models](https://mlsanity.com/paper/2303.15626/) (2023)

## Abstract (arXiv)

Countless learning tasks require dealing with sequential data. Image captioning, speech synthesis, and music generation all require that a model produce outputs that are sequences. In other domains, such as time series prediction, video analysis, and musical information retrieval, a model must learn from inputs that are sequences. Interactive tasks, such as translating natural language, engaging in dialogue, and controlling a robot, often demand both capabilities. Recurrent neural networks (RNNs) are connectionist models that capture the dynamics of sequences via cycles in the network of nodes. Unlike standard feedforward neural networks, recurrent networks retain a state that can represent information from an arbitrarily long context window. Although recurrent neural networks have traditionally been difficult to train, and often contain millions of parameters, recent advances in network architectures, optimization techniques, and parallel computation have enabled successful large-scale learning with them. In recent years, systems based on long short-term memory (LSTM) and bidirectional (BRNN) architectures have demonstrated ground-breaking performance on tasks as varied as image captioning, language translation, and handwriting recognition. In this survey, we review and synthesize the research that over the past three decades first yielded and then made practical these powerful learning models. When appropriate, we reconcile conflicting notation and nomenclature. Our goal is to provide a self-contained explication of the state of the art together with a historical perspective and references to primary research.