# Recurrent Batch Normalization

arXiv: https://arxiv.org/abs/1603.09025 · HTML: https://mlsanity.com/paper/1603.09025/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Mar 2016.

## Built on by (most cited first, 30 of 36)

- [Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges](https://mlsanity.com/paper/2104.13478/) (2021)
- [Argmax Flows and Multinomial Diffusion: Learning Categorical Distributions](https://mlsanity.com/paper/2102.05379/) (2021)
- [Low-Pass Filtering SGD for Recovering Flat Optima in the Deep Learning Optimization Landscape](https://mlsanity.com/paper/2201.08025/) (2022)
- [ScaleLong: Towards More Stable Training of Diffusion Model via Scaling Network Long Skip Connection](https://mlsanity.com/paper/2310.13545/) (2023)
- [Recurrent neural networks: vanishing and exploding gradients are not the end of the story](https://mlsanity.com/paper/2405.21064/) (2024)
- [L-SeqSleepNet: Whole-cycle Long Sequence Modelling for Automatic Sleep Staging](https://mlsanity.com/paper/2301.03441/) (2023)
- [Scalable Real-Time Recurrent Learning Using Columnar-Constructive Networks](https://mlsanity.com/paper/2302.05326/) (2023)
- [Keys to Accurate Feature Extraction Using Residual Spiking Neural Networks](https://mlsanity.com/paper/2111.05955/) (2021)
- [RF-Next: Efficient Receptive Field Search for Convolutional Neural Networks](https://mlsanity.com/paper/2206.06637/) (2022)
- [Controllable Video Captioning with an Exemplar Sentence](https://mlsanity.com/paper/2112.01073/) (2021)
- [Computing a human-like reaction time metric from stable recurrent vision models](https://mlsanity.com/paper/2306.11582/) (2023)
- [Batch Normalization Preconditioning for Neural Network Training](https://mlsanity.com/paper/2108.01110/) (2021)
- [Random orthogonal additive filters: a solution to the vanishing/exploding gradient of deep neural networks](https://mlsanity.com/paper/2210.01245/) (2022)
- [Orthogonal Gated Recurrent Unit with Neumann-Cayley Transformation](https://mlsanity.com/paper/2208.06496/) (2022)
- [Diffusing Gaussian Mixtures for Generating Categorical Data](https://mlsanity.com/paper/2303.04635/) (2023)
- [Spiking Manifesto](https://mlsanity.com/paper/2512.11843/) (2025)
- [Batch Layer Normalization, A new normalization layer for CNNs and RNN](https://mlsanity.com/paper/2209.08898/) (2022)
- [Can recurrent neural networks learn process model structure?](https://mlsanity.com/paper/2212.06430/) (2022)
- [Real-time Aerial Detection and Reasoning on Embedded-UAVs](https://mlsanity.com/paper/2305.12414/) (2023)
- [Deep Loss Convexification for Learning Iterative Models](https://mlsanity.com/paper/2411.10649/) (2024)
- [Pediatric Automatic Sleep Staging: A comparative study of state-of-the-art deep learning methods](https://mlsanity.com/paper/2108.10211/) (2021)
- [Towards Fine-grained Image Classification with Generative Adversarial Networks and Facial Landmark Detection](https://mlsanity.com/paper/2109.00891/) (2021)
- [State-Regularized Recurrent Neural Networks to Extract Automata and Explain Predictions](https://mlsanity.com/paper/2212.05178/) (2022)
- [Attention Normalization Impacts Cardinality Generalization in Slot Attention](https://mlsanity.com/paper/2407.04170/) (2024)
- [RAT+: Train Dense, Infer Sparse -- Recurrence Augmented Attention for Dilated Inference](https://mlsanity.com/paper/2602.18196/) (2026)
- [maskGRU: Tracking Small Objects in the Presence of Large Background Motions](https://mlsanity.com/paper/2201.00467/) (2022)
- [Polyphonic pitch detection with convolutional recurrent neural networks](https://mlsanity.com/paper/2202.02115/) (2022)
- [Can deep neural networks learn process model structure? An assessment framework and analysis](https://mlsanity.com/paper/2202.11985/) (2022)
- [L2BN: Enhancing Batch Normalization by Equalizing the L2 Norms of Features](https://mlsanity.com/paper/2207.02625/) (2022)
- [Breaking Time Invariance: Assorted-Time Normalization for RNNs](https://mlsanity.com/paper/2209.14439/) (2022)

## Abstract (arXiv)

We propose a reparameterization of LSTM that brings the benefits of batch normalization to recurrent neural networks. Whereas previous works only apply batch normalization to the input-to-hidden transformation of RNNs, we demonstrate that it is both possible and beneficial to batch-normalize the hidden-to-hidden transition, thereby reducing internal covariate shift between time steps. We evaluate our proposal on various sequential problems such as sequence classification, language modeling and question answering. Our empirical results show that our batch-normalized LSTM consistently leads to faster convergence and improved generalization.