# Batch Normalized Recurrent Neural Networks

arXiv: https://arxiv.org/abs/1510.01378 · HTML: https://mlsanity.com/paper/1510.01378/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Oct 2015.

## Built on by (most cited first, 15 of 15)

- [Self-Supervised Predictive Learning: A Negative-Free Method for Sound Source Localization in Visual Scenes](https://mlsanity.com/paper/2203.13412/) (2022)
- [Training Dynamical Binary Neural Networks with Equilibrium Propagation](https://mlsanity.com/paper/2103.08953/) (2021)
- [DEIR: Efficient and Robust Exploration through Discriminative-Model-Based Episodic Intrinsic Rewards](https://mlsanity.com/paper/2304.10770/) (2023)
- [Control of Dual-Sourcing Inventory Systems using Recurrent Neural Networks](https://mlsanity.com/paper/2201.06126/) (2022)
- [Batch Normalization Preconditioning for Neural Network Training](https://mlsanity.com/paper/2108.01110/) (2021)
- [Supervised Machine Learning for Effective Missile Launch Based on Beyond Visual Range Air Combat Simulations](https://mlsanity.com/paper/2207.04188/) (2022)
- [Loss Landscape Dependent Self-Adjusting Learning Rates in Decentralized Stochastic Gradient Descent](https://mlsanity.com/paper/2112.01433/) (2021)
- [A Highly Adaptive Acoustic Model for Accurate Multi-Dialect Speech Recognition](https://mlsanity.com/paper/2205.03027/) (2022)
- [Batch Layer Normalization, A new normalization layer for CNNs and RNN](https://mlsanity.com/paper/2209.08898/) (2022)
- [Deep Cellular Recurrent Network for Efficient Analysis of Time-Series Data with Spatial Information](https://mlsanity.com/paper/2101.05608/) (2021)
- [Visually-Guided Sound Source Separation with Audio-Visual Predictive Coding](https://mlsanity.com/paper/2306.10684/) (2023)
- [Attention Normalization Impacts Cardinality Generalization in Slot Attention](https://mlsanity.com/paper/2407.04170/) (2024)
- [TaLU: A Hybrid Activation Function Combining Tanh and Rectified Linear Unit to Enhance Neural Networks](https://mlsanity.com/paper/2305.04402/) (2023)
- [Federated Learning in Temporal Heterogeneity](https://mlsanity.com/paper/2309.09381/) (2023)
- [SpikingGamma: Surrogate-Gradient Free and Temporally Precise Online Training of Spiking Neural Networks with Smoothed Delays](https://mlsanity.com/paper/2602.01978/) (2026)

## Abstract (arXiv)

Recurrent Neural Networks (RNNs) are powerful models for sequential data that have the potential to learn long-term dependencies. However, they are computationally expensive to train and difficult to parallelize. Recent work has shown that normalizing intermediate representations of neural networks can significantly improve convergence rates in feedforward neural networks . In particular, batch normalization, which uses mini-batch statistics to standardize features, was shown to significantly reduce training time. In this paper, we show that applying batch normalization to the hidden-to-hidden transitions of our RNNs doesn't help the training procedure. We also show that when applied to the input-to-hidden transitions, batch normalization can lead to a faster convergence of the training criterion but doesn't seem to improve the generalization performance on both our language modelling and speech recognition tasks. All in all, applying batch normalization to RNNs turns out to be more challenging than applying it to feedforward networks, but certain variants of it can still be beneficial.