mlsanityData to 14 Oct 2026

A Theoretically Grounded Application of Dropout in Recurrent Neural Networks

arXiv 1512.05287 ↗First posted 16 Dec 2015stat.MLcited by 161

Not on the map. Why?

Abstract (arXiv)

Recurrent neural networks (RNNs) stand at the forefront of many recent developments in deep learning. Yet a major difficulty with these models is their tendency to overfit, with dropout shown to fail when applied to recurrent layers. Recent results at the intersection of Bayesian modelling and deep learning offer a Bayesian interpretation of common deep learning techniques such as dropout. This grounding of dropout in approximate Bayesian inference suggests an extension of the theoretical results, offering insights into the use of dropout with RNN models. We apply this new variational inference based dropout technique in LSTM and GRU models, assessing it on language modelling and sentiment analysis tasks. The new approach outperforms existing techniques, and to the best of our knowledge improves on the single model state-of-the-art in language modelling with the Penn Treebank (73.4 test perplexity). This extends our arsenal of variational tools in deep learning.

Key facts. Cited by 161 papers in the map (data through 14 Oct 2026). First posted on arXiv 16 Dec 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2018: 1; 2019: 0; 2020: 0; 2021: 48; 2022: 36; 2023: 30; 2024: 16; 2025: 21; 2026: 9.

161 papers.

Sort:

By area:

  1. 2021R-Drop: Regularized Dropout for Neural Networkscited by 202
  2. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learningcited by 126
  3. 2023Quantifying Uncertainty in Answers from any Language Model and Enhancing their Trustworthinesscited by 122
  4. 2021LGESQL: Line Graph Enhanced Text-to-SQL Model with Mixed Local and Non-Local Relationscited by 119
  5. 2021Stabilizing Equilibrium Models by Jacobian Regularizationcited by 72
  6. 2023Uncertainty in Natural Language Generation: From Theory to Applicationscited by 61
  7. 2023Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformerscited by 59
  8. 2021UnICORNN: A recurrent model for learning very long time dependenciescited by 56
  9. 2021Long Expressive Memory for Sequence Modelingcited by 41
  10. 2021Active Learning for Sequence Tagging with Deep Pre-trained Models and Bayesian Uncertainty Estimatescited by 39
  11. 2021AutoDropout: Learning Dropout Patterns to Regularize Deep Networkscited by 35
  12. 2021Taxonomizing local versus global structure in neural network loss landscapescited by 34
38 more
  1. 2022Estimating average causal effects from patient trajectoriescited by 29
  2. 2022Copula Conformal Prediction for Multi-step Time Series Forecastingcited by 29
  3. 2025Pre-training under infinite computecited by 29
  4. 2023Uncertainty Quantification in Machine Learning for Engineering Design and Health Prognostics: A Tutorialcited by 27
  5. 2021Selfish Sparse RNN Trainingcited by 26
  6. 2022Efficient recurrent architectures through activity sparsity and sparse back-propagation through timecited by 26
  7. 2022Exploring Predictive Uncertainty and Calibration in NLP: A Study on the Impact of Method & Data Scarcitycited by 25
  8. 2022Scientific Inference With Interpretable Machine Learning: Analyzing Models to Learn About Real-World Phenomenacited by 21
  9. 2022BayesFormer: Transformer with Uncertainty Estimationcited by 19
  10. 2022A Comprehensive Review of Digital Twin -- Part 2: Roles of Uncertainty Quantification and Optimization, a Battery Digital Twin, and Perspectivescited by 18
  11. 2021Rethinking Perturbations in Encoder-Decoders for Fast Trainingcited by 17
  12. 2022Answering Numerical Reasoning Questions in Table-Text Hybrid Contents with Graph-based Encoder and Tree-based Decodercited by 17
  13. 2023Fighting Uncertainty with Gradients: Offline Reinforcement Learning via Diffusion Score Matchingcited by 17
  14. 2023Document Flattening: Beyond Concatenating Context for Document-Level Neural Machine Translationcited by 15
  15. 2022Simple and Effective Knowledge-Driven Query Expansion for QA-Based Product Attribute Extractioncited by 12
  16. 2023Density-Softmax: Efficient Test-time Model for Uncertainty Estimation and Robustness under Distribution Shiftscited by 12
  17. 2021Continuous Learning in Neural Machine Translation using Bilingual Dictionariescited by 11
  18. 2021UniDrop: A Simple yet Effective Technique to Improve Transformer without Extra Costcited by 11
  19. 2022Conformal Methods for Quantifying Uncertainty in Spatiotemporal Data: A Surveycited by 11
  20. 2023Lewis's Signaling Game as beta-VAE For Natural Word Lengths and Segmentscited by 11
  21. 2021Low-Resource Language Modelling of South African Languagescited by 10
  22. 2021Data-driven discovery of interpretable causal relations for deep learning material laws with uncertainty propagationcited by 10
  23. 2022CipherDAug: Ciphertext based Data Augmentation for Neural Machine Translationcited by 9
  24. 2021Rethinking Data Augmentation for Low-Resource Neural Machine Translation: A Multi-Task Learning Approachcited by 8
  25. 2022ExtraPhrase: Efficient Data Augmentation for Abstractive Summarizationcited by 8
  26. 2022Bi-SimCut: A Simple Strategy for Boosting Neural Machine Translationcited by 8
  27. 2023Bayesian neural networks via MCMC: a Python-based tutorialcited by 8
  28. 2021SpliceOut: A Simple and Efficient Audio Augmentation Methodcited by 7
  29. 2023Ensembled Prediction Intervals for Causal Outcomes Under Hidden Confoundingcited by 7
  30. 2024Density-Regression: Efficient and Distance-Aware Deep Regressor for Uncertainty Estimation under Distribution Shiftscited by 7
  31. 2024Uncertainty-boosted Robust Video Activity Anticipationcited by 7
  32. 2024Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modelingcited by 7
  33. 2024Learning charges and long-range interactions from energies and forcescited by 7
  34. 2018Dropout Regularization in Hierarchical Mixture of Expertscited by 6
  35. 2021Active2 Learning: Actively reducing redundancies in Active Learning methods for Sequence Tagging and Machine Translationcited by 6
  36. 2021Enhancing predictive skills in physically-consistent way: Physics Informed Machine Learning for Hydrological Processescited by 6
  37. 2021Tensor-Train Recurrent Neural Networks for Interpretable Multi-Way Financial Forecastingcited by 6
  38. 2021Deep Visual Navigation under Partial Observabilitycited by 6

50 most cited of 161 All 161 (JSON)