# Recurrent Dropout without Memory Loss

arXiv: https://arxiv.org/abs/1603.05118 · HTML: https://mlsanity.com/paper/1603.05118/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Mar 2016.

## Built on by (most cited first, 22 of 22)

- [R-Drop: Regularized Dropout for Neural Networks](https://mlsanity.com/paper/2106.14448/) (2021)
- [Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers](https://mlsanity.com/paper/2303.01610/) (2023)
- [LoRA Dropout as a Sparsity Regularizer for Overfitting Control](https://mlsanity.com/paper/2404.09610/) (2024)
- [Spiking Neural Networks with Improved Inherent Recurrence Dynamics for Sequential Learning](https://mlsanity.com/paper/2109.01905/) (2021)
- [Gating Dropout: Communication-efficient Regularization for Sparsely Activated Transformers](https://mlsanity.com/paper/2205.14336/) (2022)
- [Learning Hierarchical Structures with Differentiable Nondeterministic Stacks](https://mlsanity.com/paper/2109.01982/) (2021)
- [Deep Visual Navigation under Partial Observability](https://mlsanity.com/paper/2109.07752/) (2021)
- [A Bayesian approach to quantifying uncertainties and improving generalizability in traffic prediction models](https://mlsanity.com/paper/2307.05946/) (2023)
- [A Survey on Dropout Methods and Experimental Verification in Recommendation](https://mlsanity.com/paper/2204.02027/) (2022)
- [Recency Dropout for Recurrent Recommender Systems](https://mlsanity.com/paper/2201.11016/) (2022)
- [Neuroplasticity in Artificial Intelligence -- An Overview and Inspirations on Drop In & Out Learning](https://mlsanity.com/paper/2503.21419/) (2025)
- [Fine-tuning Handwriting Recognition systems with Temporal Dropout](https://mlsanity.com/paper/2102.00511/) (2021)
- [How Do Neural Sequence Models Generalize? Local and Global Context Cues for Out-of-Distribution Prediction](https://mlsanity.com/paper/2111.03108/) (2021)
- [Towards an AI-based Early Warning System for Bridge Scour](https://mlsanity.com/paper/2208.10500/) (2022)
- [Gates Are Not What You Need in RNNs](https://mlsanity.com/paper/2108.00527/) (2021)
- [Deep Neural Network Calibration by Reducing Classifier Shift with Stochastic Masking](https://mlsanity.com/paper/2508.09116/) (2025)
- [Structured in Space, Randomized in Time: Leveraging Dropout in RNNs for Efficient Training](https://mlsanity.com/paper/2106.12089/) (2021)
- [Acoustic characterization of speech rhythm: going beyond metrics with recurrent neural networks](https://mlsanity.com/paper/2401.14416/) (2024)
- [Contrast-Enhanced Gating in GRUs for Robust Low-Data Sequence Learning](https://mlsanity.com/paper/2402.09034/) (2024)
- [Self Identity Mapping](https://mlsanity.com/paper/2509.18165/) (2025)
- [Distribution-Aware Conformal Prediction: A Framework for generating efficient prediction intervals for time series](https://mlsanity.com/paper/2605.26569/) (2026)
- [Loop Dropout: Regularizing Shared Updates in Looped Language Models](https://mlsanity.com/paper/2609.34218/) (2026)

## Abstract (arXiv)

This paper presents a novel approach to recurrent neural network (RNN) regularization. Differently from the widely adopted dropout method, which is applied to textitforward connections of feed-forward architectures or RNNs, we propose to drop neurons directly in textitrecurrent connections in a way that does not cause loss of long-term memory. Our approach is as easy to implement and apply as the regular feed-forward dropout and we demonstrate its effectiveness for Long Short-Term Memory network, the most popular type of RNN cells. Our experiments on NLP benchmarks show consistent improvements even when combined with conventional feed-forward dropout.