mlsanityData to 7 Oct 2026

Recurrent Dropout without Memory Loss

arXiv 1603.05118 ↗First posted 16 Mar 2016cs.CLcited by 22

Not on the map. Why?

Abstract (arXiv)

This paper presents a novel approach to recurrent neural network (RNN) regularization. Differently from the widely adopted dropout method, which is applied to textitforward connections of feed-forward architectures or RNNs, we propose to drop neurons directly in textitrecurrent connections in a way that does not cause loss of long-term memory. Our approach is as easy to implement and apply as the regular feed-forward dropout and we demonstrate its effectiveness for Long Short-Term Memory network, the most popular type of RNN cells. Our experiments on NLP benchmarks show consistent improvements even when combined with conventional feed-forward dropout.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 Mar 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 4; 2023: 2; 2024: 3; 2025: 3; 2026: 2.

22 papers.

Sort:

By area:

  1. 2021R-Drop: Regularized Dropout for Neural Networkscited by 202
  2. 2023Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformerscited by 59
  3. 2024LoRA Dropout as a Sparsity Regularizer for Overfitting Controlcited by 35
  4. 2021Spiking Neural Networks with Improved Inherent Recurrence Dynamics for Sequential Learningcited by 27
  5. 2022Gating Dropout: Communication-efficient Regularization for Sparsely Activated Transformerscited by 22
  6. 2021Learning Hierarchical Structures with Differentiable Nondeterministic Stackscited by 16
  7. 2021Deep Visual Navigation under Partial Observabilitycited by 6
  8. 2023A Bayesian approach to quantifying uncertainties and improving generalizability in traffic prediction modelscited by 5
  9. 2022A Survey on Dropout Methods and Experimental Verification in Recommendationcited by 4
  10. 2022Recency Dropout for Recurrent Recommender Systemscited by 3
  11. 2025Neuroplasticity in Artificial Intelligence -- An Overview and Inspirations on Drop In & Out Learningcited by 3
  12. 2021Fine-tuning Handwriting Recognition systems with Temporal Dropoutcited by 2
10 more
  1. 2021How Do Neural Sequence Models Generalize? Local and Global Context Cues for Out-of-Distribution Predictioncited by 2
  2. 2022Towards an AI-based Early Warning System for Bridge Scourcited by 2
  3. 2021Gates Are Not What You Need in RNNscited by 1
  4. 2025Deep Neural Network Calibration by Reducing Classifier Shift with Stochastic Maskingcited by 1
  5. 2021Structured in Space, Randomized in Time: Leveraging Dropout in RNNs for Efficient Trainingcited by 0
  6. 2024Acoustic characterization of speech rhythm: going beyond metrics with recurrent neural networkscited by 0
  7. 2024Contrast-Enhanced Gating in GRUs for Robust Low-Data Sequence Learningcited by 0
  8. 2025Self Identity Mappingcited by 0
  9. 2026Distribution-Aware Conformal Prediction: A Framework for generating efficient prediction intervals for time seriescited by 0
  10. 2026Loop Dropout: Regularizing Shared Updates in Looped Language Modelscited by 0