# Recurrent Neural Network Regularization

arXiv: https://arxiv.org/abs/1409.2329 · HTML: https://mlsanity.com/paper/1409.2329/

## Key facts

- Cited by 189 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Sep 2014.

## Built on by (most cited first, 30 of 189)

- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [ChemRL-GEM: Geometry Enhanced Molecular Representation Learning for Property Prediction](https://mlsanity.com/paper/2106.06130/) (2021)
- [A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations](https://mlsanity.com/paper/2308.06767/) (2023)
- [Large Language Models for Robotics: A Survey](https://mlsanity.com/paper/2311.07226/) (2023)
- [Urban Driver: Learning to Drive from Real-world Demonstrations Using Policy Gradients](https://mlsanity.com/paper/2109.13333/) (2021)
- [Trustworthy AI: A Computational Perspective](https://mlsanity.com/paper/2107.06641/) (2021)
- [CAT: Cross Attention in Vision Transformer](https://mlsanity.com/paper/2106.05786/) (2021)
- [STream3R: Scalable Sequential 3D Reconstruction with Causal Transformer](https://mlsanity.com/paper/2508.10893/) (2025)
- [BERT is to NLP what AlexNet is to CV: Can Pre-Trained Language Models Identify Analogies?](https://mlsanity.com/paper/2105.04949/) (2021)
- [Learning from Teaching Regularization: Generalizable Correlations Should be Easy to Imitate](https://mlsanity.com/paper/2402.02769/) (2024)
- [Noisy Recurrent Neural Networks](https://mlsanity.com/paper/2102.04877/) (2021)
- [Dissecting Transformer Length Extrapolation via the Lens of Receptive Field Analysis](https://mlsanity.com/paper/2212.10356/) (2022)
- [When Large Language Models Meet Vector Databases: A Survey](https://mlsanity.com/paper/2402.01763/) (2024)
- [Visual Attention Methods in Deep Learning: An In-Depth Survey](https://mlsanity.com/paper/2204.07756/) (2022)
- [Learning a Prior over Intent via Meta-Inverse Reinforcement Learning](https://mlsanity.com/paper/1805.12573/) (2018)
- [AutoDropout: Learning Dropout Patterns to Regularize Deep Networks](https://mlsanity.com/paper/2101.01761/) (2021)
- [Recent advances for quantum classifiers](https://mlsanity.com/paper/2108.13421/) (2021)
- [Sparse Meets Dense: Unified Generative Recommendations with Cascaded Sparse-Dense Representations](https://mlsanity.com/paper/2503.02453/) (2025)
- [Multi-Agent Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2408.09675/) (2024)
- [Vision-Based Traffic Accident Detection and Anticipation: A Survey](https://mlsanity.com/paper/2308.15985/) (2023)
- [3D Human Motion Prediction: A Survey](https://mlsanity.com/paper/2203.01593/) (2022)
- [Attentive Neural Controlled Differential Equations for Time-series Classification and Forecasting](https://mlsanity.com/paper/2109.01876/) (2021)
- [Topology-aware Convolutional Neural Network for Efficient Skeleton-based Action Recognition](https://mlsanity.com/paper/2112.04178/) (2021)
- [Pre-training under infinite compute](https://mlsanity.com/paper/2509.14786/) (2025)
- [Positive-Negative Momentum: Manipulating Stochastic Gradient Noise to Improve Generalization](https://mlsanity.com/paper/2103.17182/) (2021)
- [Faithfulness in Natural Language Generation: A Systematic Survey of Analysis, Evaluation and Optimization Methods](https://mlsanity.com/paper/2203.05227/) (2022)
- [Selfish Sparse RNN Training](https://mlsanity.com/paper/2101.09048/) (2021)
- [PQLM -- Multilingual Decentralized Portable Quantum Language Model for Privacy Protection](https://mlsanity.com/paper/2210.03221/) (2022)
- [Exploring Predictive Uncertainty and Calibration in NLP: A Study on the Impact of Method & Data Scarcity](https://mlsanity.com/paper/2210.15452/) (2022)

## Abstract (arXiv)

We present a simple regularization technique for Recurrent Neural Networks (RNNs) with Long Short-Term Memory (LSTM) units. Dropout, the most successful technique for regularizing neural networks, does not work well with RNNs and LSTMs. In this paper, we show how to correctly apply dropout to LSTMs, and show that it substantially reduces overfitting on a variety of tasks. These tasks include language modeling, speech recognition, image caption generation, and machine translation.