# Regularizing RNNs by Stabilizing Activations

arXiv: https://arxiv.org/abs/1511.08400 · HTML: https://mlsanity.com/paper/1511.08400/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Nov 2015.

## Built on by (most cited first, 6 of 6)

- [Sparsely Changing Latent States for Prediction and Planning in Partially Observable Domains](https://mlsanity.com/paper/2110.15949/) (2021)
- [All That Glitters is Not Novel: Plagiarism in AI Generated Research](https://mlsanity.com/paper/2502.16487/) (2025)
- [Recurrent Neural Network from Adder's Perspective: Carry-lookahead RNN](https://mlsanity.com/paper/2106.12901/) (2021)
- [Oja's plasticity rule overcomes several challenges of training neural networks under biological constraints](https://mlsanity.com/paper/2408.08408/) (2024)
- [Look Backward and Forward: Self-Knowledge Distillation with Bidirectional Decoder for Neural Machine Translation](https://mlsanity.com/paper/2203.05248/) (2022)
- [An Adaptive Method Stabilizing Activations for Enhanced Generalization](https://mlsanity.com/paper/2506.08353/) (2025)

## Abstract (arXiv)

We stabilize the activations of Recurrent Neural Networks (RNNs) by penalizing the squared distance between successive hidden states' norms. This penalty term is an effective regularizer for RNNs including LSTMs and IRNNs, improving performance on character-level language modeling and phoneme recognition, and outperforming weight noise and dropout. We achieve competitive performance (18.6% PER) on the TIMIT phoneme recognition task for RNNs evaluated without beam search or an RNN transducer. With this penalty term, IRNN can achieve similar performance to LSTM on language modeling, although adding the penalty term to the LSTM results in superior performance. Our penalty term also prevents the exponential growth of IRNN's activations outside of their training horizon, allowing them to generalize to much longer sequences.