# Noisin: Unbiased Regularization for Recurrent Neural Networks

arXiv: https://arxiv.org/abs/1805.01500 · HTML: https://mlsanity.com/paper/1805.01500/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 May 2018.

## Built on by (most cited first, 8 of 8)

- [Noisy Recurrent Neural Networks](https://mlsanity.com/paper/2102.04877/) (2021)
- [Asymmetric Heavy Tails and Implicit Bias in Gaussian Noise Injections](https://mlsanity.com/paper/2102.07006/) (2021)
- [Tensor-Train Recurrent Neural Networks for Interpretable Multi-Way Financial Forecasting](https://mlsanity.com/paper/2105.04983/) (2021)
- [One Step Forward and K Steps Back: Better Reasoning with Denoising Recursion Models](https://mlsanity.com/paper/2604.18839/) (2026)
- [State-Regularized Recurrent Neural Networks to Extract Automata and Explain Predictions](https://mlsanity.com/paper/2212.05178/) (2022)
- [On the Regularization of Learnable Embeddings for Time Series Forecasting](https://mlsanity.com/paper/2410.14630/) (2024)
- [Overlap-Adaptive Regularization for Conditional Average Treatment Effect Estimation](https://mlsanity.com/paper/2509.24962/) (2025)
- [Deep Probabilistic Graphical Modeling](https://mlsanity.com/paper/2104.12053/) (2021)

## Abstract (arXiv)

Recurrent neural networks (RNNs) are powerful models of sequential data. They have been successfully used in domains such as text and speech. However, RNNs are susceptible to overfitting; regularization is important. In this paper we develop Noisin, a new method for regularizing RNNs. Noisin injects random noise into the hidden states of the RNN and then maximizes the corresponding marginal likelihood of the data. We show how Noisin applies to any RNN and we study many different types of noise. Noisin is unbiased--it preserves the underlying RNN on average. We characterize how Noisin regularizes its RNN both theoretically and empirically. On language modeling benchmarks, Noisin improves over dropout by as much as 12.2% on the Penn Treebank and 9.4% on the Wikitext-2 dataset. We also compared the state-of-the-art language model of Yang et al. 2017, both with and without Noisin. On the Penn Treebank, the method with Noisin more quickly reaches state-of-the-art performance.