mlsanityData to 7 Oct 2026

Recurrent Batch Normalization

arXiv 1603.09025 ↗First posted 30 Mar 2016cs.LGcited by 35

Not on the map. Why?

Abstract (arXiv)

We propose a reparameterization of LSTM that brings the benefits of batch normalization to recurrent neural networks. Whereas previous works only apply batch normalization to the input-to-hidden transformation of RNNs, we demonstrate that it is both possible and beneficial to batch-normalize the hidden-to-hidden transition, thereby reducing internal covariate shift between time steps. We evaluate our proposal on various sequential problems such as sequence classification, language modeling and question answering. Our empirical results show that our batch-normalized LSTM consistently leads to faster convergence and improved generalization.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). First posted on arXiv 30 Mar 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 13; 2023: 8; 2024: 3; 2025: 3; 2026: 1.

35 papers.

Sort:

By area:

  1. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,131
  2. 2021Argmax Flows and Multinomial Diffusion: Learning Categorical Distributionscited by 566
  3. 2022Low-Pass Filtering SGD for Recovering Flat Optima in the Deep Learning Optimization Landscapecited by 39
  4. 2023ScaleLong: Towards More Stable Training of Diffusion Model via Scaling Network Long Skip Connectioncited by 25
  5. 2024Recurrent neural networks: vanishing and exploding gradients are not the end of the storycited by 25
  6. 2023L-SeqSleepNet: Whole-cycle Long Sequence Modelling for Automatic Sleep Stagingcited by 21
  7. 2023Scalable Real-Time Recurrent Learning Using Columnar-Constructive Networkscited by 11
  8. 2021Keys to Accurate Feature Extraction Using Residual Spiking Neural Networkscited by 10
  9. 2022RF-Next: Efficient Receptive Field Search for Convolutional Neural Networkscited by 10
  10. 2021Controllable Video Captioning with an Exemplar Sentencecited by 9
  11. 2023Computing a human-like reaction time metric from stable recurrent vision modelscited by 6
  12. 2021Batch Normalization Preconditioning for Neural Network Trainingcited by 5
23 more
  1. 2022Random orthogonal additive filters: a solution to the vanishing/exploding gradient of deep neural networkscited by 5
  2. 2022Orthogonal Gated Recurrent Unit with Neumann-Cayley Transformationcited by 4
  3. 2023Diffusing Gaussian Mixtures for Generating Categorical Datacited by 4
  4. 2022Batch Layer Normalization, A new normalization layer for CNNs and RNNcited by 3
  5. 2022Can recurrent neural networks learn process model structure?cited by 3
  6. 2023Real-time Aerial Detection and Reasoning on Embedded-UAVscited by 3
  7. 2024Deep Loss Convexification for Learning Iterative Modelscited by 3
  8. 2025Spiking Manifestocited by 3
  9. 2021Pediatric Automatic Sleep Staging: A comparative study of state-of-the-art deep learning methodscited by 2
  10. 2021Towards Fine-grained Image Classification with Generative Adversarial Networks and Facial Landmark Detectioncited by 2
  11. 2022State-Regularized Recurrent Neural Networks to Extract Automata and Explain Predictionscited by 2
  12. 2026RAT+: Train Dense, Infer Sparse -- Recurrence Augmented Attention for Dilated Inferencecited by 2
  13. 2022maskGRU: Tracking Small Objects in the Presence of Large Background Motionscited by 1
  14. 2022Polyphonic pitch detection with convolutional recurrent neural networkscited by 1
  15. 2022Can deep neural networks learn process model structure? An assessment framework and analysiscited by 1
  16. 2024Attention Normalization Impacts Cardinality Generalization in Slot Attentioncited by 1
  17. 2022L2BN: Enhancing Batch Normalization by Equalizing the L2 Norms of Featurescited by 0
  18. 2022Breaking Time Invariance: Assorted-Time Normalization for RNNscited by 0
  19. 2022KL Regularized Normalization Framework for Low Resource Taskscited by 0
  20. 2023Federated Learning in Temporal Heterogeneitycited by 0
  21. 2023Unified Batch Normalization: Identifying and Alleviating the Feature Condensation in Batch Normalization and a Unified Frameworkcited by 0
  22. 2025Diffusion Denoiser-Aided Gyrocompassingcited by 0
  23. 2025Learnability Window in Gated Recurrent Neural Networkscited by 0