mlsanityData to 14 Oct 2026

Online Normalization for Training Neural Networks

arXiv 1905.05894 ↗First posted 15 May 2019cs.LG stat.MLcited by 29

Not on the map. Why?

Abstract (arXiv)

Online Normalization is a new technique for normalizing the hidden activations of a neural network. Like Batch Normalization, it normalizes the sample dimension. While Online Normalization does not use batches, it is as accurate as Batch Normalization. We resolve a theoretical limitation of Batch Normalization by introducing an unbiased technique for computing the gradient of normalized activations. Online Normalization works with automatic differentiation by adding statistical normalization as a primitive. This technique can be used in cases not covered by some other normalizers, such as recurrent networks, fully connected networks, and networks with activation memory requirements prohibitive for batching. We show its applications to image classification, image segmentation, and language modeling. We present formal proofs and experimental results on ImageNet, CIFAR, and PTB datasets.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). First posted on arXiv 15 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 9; 2022: 7; 2023: 5; 2024: 1; 2025: 4; 2026: 3.

29 papers.

Sort:

  1. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  2. 2021Continual Backprop: Stochastic Gradient Descent with Persistent Randomnesscited by 80
  3. 2023Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networkscited by 53
  4. 2023Maintaining Plasticity in Deep Continual Learningcited by 43
  5. 2025Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-trainingcited by 43
  6. 2021Beyond BatchNorm: Towards a Unified Understanding of Normalization in Deep Learningcited by 25
  7. 2024Analyzing & Reducing the Need for Learning Rate Warmup in GPT Trainingcited by 22
  8. 2021On the Periodic Behavior of Neural Network Training with Batch Normalization and Weight Decaycited by 20
  9. 2025Weight Decay may matter more than muP for Learning Rate Transfer in Practicecited by 20
  10. 2022Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimescited by 19
  11. 2021Momentum^2 Teacher: Momentum Teacher with Momentum Statistics for Self-Supervised Learningcited by 14
  12. 2022Toward Equation of Motion for Deep Neural Networks: Continuous-time Gradient Descent and Discretization Error Analysiscited by 13
17 more
  1. 2021Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependencecited by 12
  2. 2022Understanding the Failure of Batch Normalization for Transformers in NLPcited by 12
  3. 2025Training Dynamics of the Cooldown Stage in Warmup-Stable-Decay Learning Rate Schedulercited by 12
  4. 2021Stochastic Whitening Batch Normalizationcited by 9
  5. 2022Delving into the Estimation Shift of Batch Normalization in a Networkcited by 7
  6. 2021Making EfficientNet More Efficient: Exploring Batch-Independent Normalization, Group Convolutions and Reduced Resolution Trainingcited by 6
  7. 2022RevBiFPN: The Fully Reversible Bidirectional Feature Pyramid Networkcited by 6
  8. 2023Neural Characteristic Activation Analysis and Geometric Parameterization for ReLU Networkscited by 5
  9. 2021Deep Unitary Convolutional Neural Networkscited by 2
  10. 2023Ghost Noise for Regularizing Deep Neural Networkscited by 1
  11. 2025AlphaGrad: Non-Linear Gradient Normalization Optimizercited by 1
  12. 2026Weight-norm Criticality: A Mechanism for Loss Spikes Induced by the Normalization and Weight Decaycited by 1
  13. 2021A Projection Algorithm for the Unitary Weightscited by 0
  14. 2022SML:Enhance the Network Smoothness with Skip Meta Logit for CTR Predictioncited by 0
  15. 2023Unified Batch Normalization: Identifying and Alleviating the Feature Condensation in Batch Normalization and a Unified Frameworkcited by 0
  16. 2026ZeroDVFS: Zero-Shot LLM-Guided Core and Frequency Allocation for Embedded Platformscited by 0
  17. 2026When Does Scale-Invariant Optimization Become Unstable? An Exact Schedule Law with Weight Decaycited by 0